Have we hit the limit of Scaling Laws and a plateau in AI progress?

Published on

Have you ever heard of AI “Scaling Laws”? They’re one of the most important ideas and, at the same time, surprisingly simple, that guided the revolution we’re living through.

Think of them as the “cake recipe” the big labs have been following for the past few years, formalized in a 2020 paper from OpenAI called "Scaling Laws for Neural Language Models".

The discovery of Scaling Laws

The key finding was that model improvement followed an empirical “law”, discovered through lots of internal experiments: by increasing computing power (compute, meaning the total amount of math operations used in training - FLOPs), the amount of training data and the size of the model’s neural network (number of parameters), the performance of language models improves in a predictable way.

After that paper came out, the dominant view became, basically, that if you have more computing power, it’s worth putting almost all of it into growing the number of parameters of the model’s neural network. That’s how models like GPT-3 (175B parameters) were born: really big, but trained on relatively few tokens (~300 billion).

From there, the bet of almost every big player was just one: “SCALE IS ALL YOU NEED”.

Scale is all you need

And for a good while, that bet paid off beautifully. Models like the GPT series not only got better at tasks they already knew, but also supposedly started developing “emergent abilities” (capabilities nobody trained them directly to do, but that they managed to learn through correlations), like doing math (even if not perfectly), reasoning step by step (the famous “Chain-of-Thought”) or understanding the meaning of a word from context, simply because they got gigantic.

However, still back then, OpenAI supposedly found out that the original recipe from the paper wasn’t exactly wrong, but it was a little unbalanced. If you split the same compute budget in a more balanced way between network parameters and amount of data, you get much better results.

In other words, in practice, if you only increase the number of parameters but not the amount of data, the model will “memorize” and overfit. If you only increase the amount of data but keep the model’s neural network small, it won’t be able to learn enough. If you only increase compute without growing the model or the amount of data, it’s kind of a waste of resources.

They supposedly fixed this internally, but without telling anyone (after all, it could give them a competitive edge over the competition).

Chinchilla, and the rediscovery

In 2022, Google DeepMind also managed to find this imbalance, and published the paper “Training Compute-Optimal Large Language Models” about a model called “Chinchilla” that, with “just” 70 billion parameters (but trained on 1.4 trillion tokens), was able to beat much bigger models, like Gopher (280 billion parameters), with the same compute budget, and had a much lower inference cost. In other words, many of the gigantic models companies were training were being “underused” because they didn’t have enough training data.

Chinchilla

From then on, the new golden rule became: every time you increase the size of the model’s neural network, you also need to increase the amount of data used in training.

According to them, the ideal number of training tokens, aiming to optimize training cost, should be about 20 times the number of parameters of the model. So, thinking about the most recent models, in theory, a 27-billion-parameter Gemma 3 from Google should be trained on ~540 billion tokens, and OpenAI’s GPT-OSS-120B on ~2.4 trillion tokens.

In practice, the biggest models we use, like GPT-5, Gemini 2.5 Pro and Claude Opus 4, were trained on something between 14 and 18 trillion tokens (numbers not confirmed by the companies, but estimated by researchers). One public case is DeepSeek-V3, which was trained on ~14.8 trillion tokens.

Of course, since it’s from 2022, that paper doesn’t take into account more recent developments in how these models are trained, like the concept of Mixture-of-Experts, which is when you don’t train a model as one single dense neural network, where all the parameters are active all the time, but rather as a set of sub-networks inside the same model, where only a fraction of them is actually activated, and a router decides which experts are most relevant to solve that type of problem. That changes the dynamics of scale, allowing models with a gigantic total number of parameters but a reduced computational cost, since what matters becomes the number of active parameters.

So now what? Have we hit a plateau?

Fast-forwarding to today, the big conversation in the AI world is whether this golden age of scale is coming to an end. The inside gossip from some of the big AI labs would be that these performance gains from scaling alone are shrinking, and a lot of people speculate we may have reached a plateau.

Plateau

Some cutting-edge models, like GPT-5 and Llama 4, even though they improved over their predecessors, didn’t show a performance jump as big as the one we saw from GPT-3 to GPT-4, even with investments orders of magnitude larger.

Some of the main arguments for this possible plateau are:

  • Running out of high-quality data: we’ve already used the entire internet to train every top-tier model out there. Ilya Sutskever himself, one of the founders of OpenAI and now the founder of Safe Superintelligence, said at the NeurIPS 2024 conference that “pre-training as we know it will end” because, basically, “we only have one internet”. The obvious solution would be to use synthetic data generated by other AIs, but that can lead to a known risk called “model collapse” or “AI dementia”, where the model would start learning from its own biases and its quality would steadily degrade in a vicious circle (if you want to know more about synthetic data, I REALLY recommend the episode of the IA Sob Controle podcast I recorded with Marah Abdin, head of synthetic data at Microsoft).

  • Diminishing returns: the cost to train these models is already in the hundreds of millions of dollars and heading toward billions (remember the Stargate Project, a private consortium with OpenAI and other companies and political backing from the US government, with investments of up to 500 billion dollars in AI infrastructure? And Elon Musk’s xAI Colossus data center with 200k GPUs?). The problem is that the performance return no longer seems to justify that absurd cost. Spending 10x more to get an improvement of just 5% starts to not make much economic sense.

  • Architecture limits: the Transformer architecture, which is the basis of all this, most likely isn’t the most efficient one for scaling infinitely. We may be reaching a point where the architecture itself becomes a bottleneck, and that’s something people like Yann LeCun, one of the fathers of Deep Learning, have been saying for a long time. There are new proposals, like diffusion models (common in images) being used for text, and Energy-Based Transformers, but we don’t have anything very definitive or certain on the horizon yet.

Of course, that’s not a unanimous opinion. Dario Amodei, CEO of Anthropic, for example, still believes scaling laws will keep working. Andre Saraiva, a Brazilian who works at OpenAI and took part in training the o1, o3 and GPT-5 models, whom I also interviewed recently, said in the IA Sob Controle episode about the GPT-5 launch that, in his view, the path to an AI comparable to humans still lies in scaling laws.

With that uncertainty in the air, the race now is to “scale smarter”, something we’ve seen the folks at DeepSeek AI do very well, especially because they don’t have access to the best GPUs on the market (which are NVIDIA’s), due to restrictions on purchases by Chinese companies imposed by the US government.

If simply increasing everything is no longer the answer, innovation is moving to other areas. Some ideas every company implemented to keep improving their models were:

  • A mix of Mixture-of-Experts with reasoning at inference time, where the model uses more computing power during inference, internally drafting a step-by-step, a chain of thought, before it starts giving you the final answer;

  • Data pruning, which is when you don’t throw all the raw data you have available at the model, but make a careful selection to raise its quality, removing duplicates, low-quality data (like spam, bad code) and applying automatic filters with other models. Studies show that it’s possible to train models with just half of the original data and get equal or even better performance;

  • Tool calling, making the user experience more “agentic”, making it possible to interpret and run Python code right in the chat window, generate and display charts and web pages, etc.

  • Multimodality, with image understanding and generation, etc.

Still, it’s worth mentioning an interesting counterpoint: the fine-tuned versions of o3 and Gemini 2.5 Pro that managed to win the gold medal at the International Mathematical Olympiad in 2025, something unexpected even for the researchers themselves.

I don’t have a 100% formed opinion on the topic yet, but I tend to believe that we are, yes, on a plateau, and that the only thing that will get us out of it is a relevant innovation in model architecture, not just “throwing more compute at it”. Another thing that could take us to the next step is optimizing training and inference with GPUs, TPUs and new AI-focused chips that several companies are trying to develop, in case there’s some breakthrough that makes these processes an order of magnitude cheaper.

The AI race is far from over, but the rules of the game are, without a doubt, changing.

And you, what do you think? Have we hit the limit of scale or not?