New Yann LeCun paper - Transformers without Normalization
Published on
A new AI paper came out from Yann LeCun, one of the fathers of Deep Learning, and it could bring a radical change to the architecture of deep neural networks and LLMs.
The paper is called “Transformers without Normalization” and presents a surprisingly simple technique called Dynamic Tanh (DyT), which replaces the traditional normalization layers (Layer Norm or RMSNorm) with a single operation: DyT(x) = tanh(αx), a beefier hyperbolic tangent.
Quick context: currently, Transformers are a dominant architecture in AI, especially in language processing and computer vision. Normalization layers (LayerNorm, RMSNorm, among others) are traditionally used to stabilize and speed up training, making sure the values of the network’s activations don’t become too extreme, which could make it hard for the model to learn.
But now, DyT suggests replacing those traditional layers with a much simpler operation: DyT(x) = tanh(αx), where α is a trainable parameter. The insight they had to get there was pretty interesting: they noticed that normalization layers, deep down, end up kind of “squeezing” the data in a way that resembled the tanh function (hyperbolic tangent). So they thought: “why not use a tanh function that we can control?”. The α parameter could then be learned during training, dynamically adjusting the scale of the values. And wouldn’t you know it, it worked!
DyT keeps the main benefits of normalization, like compressing extreme values, but without having to compute means and standard deviations during training, significantly reducing computational complexity and speeding up the model.
And the most interesting part is that, according to the paper’s results, this new Transformer architecture with DyT managed to match or beat the performance of the old architecture with normalization, without needing hyperparameter adjustments in many cases, and considerably speeding up training and inference time compared to models using traditional normalization layers (up to 52% faster in some cases).
It’s worth reading the paper to understand the details (there’s a lot of technical stuff, but that’s the central idea).
- Link to the paper: https://arxiv.org/abs/2503.10622