Energy-Based Transformers: Toward AI Models That Know When They Don't Know

Published on

A new AI paper just came out, one of the most interesting I’ve seen lately (and Yann LeCun (@ylecun), one of the fathers of Deep Learning, agrees).

In the famous book “Thinking, Fast and Slow”, Daniel Kahneman popularized an idea from psychology, that we have two thinking systems: System 1 (fast, like dodging an obstacle) and System 2 (slow, used for solving complex problems or planning).

By analogy, today’s AIs work with System 1. They generate text just by predicting the next word (token) (or all of them at once with diffusion), but when it comes to System 2, things get complicated.

With that inspiration, the researchers wondered: is it possible to teach an AI to “think slowly” natively, using only unsupervised learning, without needing constant supervision or tricks (like “reasoning” with Chain-of-Thought)?

That’s how they arrived at “Energy-Based Transformers (EBTs)”, a new architecture that could be a real turning point.

The idea is this: instead of training the model to predict the next token, like a standard LLM, they train it to be kind of a verifier. It learns to give a score (called “Energy”) to any “question + possible answer” pair, where good answers get low energy and bad answers get high energy.

So the model’s “thinking” process basically becomes an optimization process (a gradient descent on the “Energy”). It starts with a random answer and adjusts it iteratively until it finds the one with the lowest energy (the valley of the gradient).

And the results are REALLY impressive. According to them, EBTs were up to 35% more efficient during pre-training compared with standard feed-forward Transformers. At large scale, it could be even more.

On top of that, unlike standard Transformers, EBTs improve their performance the more “time” (optimization steps) they have to “think”. The paper shows gains of up to 29% on language tasks and up to 99% fewer steps on image tasks (denoising).

In an emergent way, EBTs developed a notion of “uncertainty”. For easy problems, they reach a low-energy answer really fast. For complex problems, the energy stays high for longer, which suggests the model “knows that it doesn’t know” and keeps searching.

And maybe the coolest point: EBTs generalized better to new tasks, and the gain from “thinking more” is even bigger on data very different from the training data (out-of-distribution), which is exactly where robust reasoning matters most.

In practice, it’s a potential new architecture that could bring a paradigm shift. Instead of focusing only on models that “generate”, the future may lie in models that learn to “verify” and, from there, build their answers iteratively. That makes reasoning something more native to the architecture, and not a trick applied on top.

Here’s the link to the paper.