Scaling Reinforcement Learning in LLMs

Published on

One of the areas I think has the most potential in AI training is Reinforcement Learning, which will be one of the study areas in my master’s. One of its “fathers” is Richard Sutton, the latest Turing Award winner (the Nobel of computing), and it’s an area that gained a lot of strength with Google DeepMind’s AlphaGo (which has an EXCELLENT documentary on YouTube), and more recently, with ALL the LLMs we use.

The problem is that, despite all this success, training LLMs with RL has always been more “art” than “science”, in the sense that a lot of what was done was pretty much “trial and error”. Well, this week a paper came out that might change that story.

Meta just published a paper called “The Art of Scaling Reinforcement Learning”, which attacks exactly one of the biggest problems in model post-training today. Until now, scaling RL training was kind of a shot in the dark. You invested thousands of GPU hours without being sure whether your training “recipe” would really give good results at scale, or whether it would simply hit a performance ceiling and stagnate.

Reinforcement Learning at scale

What the Meta researchers did was basically create a framework to try to predict that performance, something like the “Scaling Laws” we already know for model pre-training, but for Reinforcement Learning. They spent more than 400,000 GPU hours on tests, and managed to find a way to model RL’s performance gains in a predictable way!

Some conclusions they drew:

  • they propose using sigmoid curves (shaped like an “S”) to accurately predict how the model will behave and where it will stagnate.

  • methods that look great with little compute can be worse at large scale. The paper shows that some approaches start slower, but reach a much higher performance “ceiling” in the end. You can’t rely on small tests alone.

  • many common optimizations only affect efficiency (how fast the model learns), but not the performance ceiling (how good it can get). The most critical decisions are the ones that raise that ceiling, like the type of algorithm, the loss function (they point to CISPO as very robust), and even the precision of the calculations (using FP32 on the logits).

The “recipe” they created (and called “ScaleRL”) doesn’t invent anything totally new, but it combines the best existing techniques in a way that proved more stable and scalable. In their tests, it beat other famous recipes, like the ones used by DeepSeek and Qwen.

In practice, this has the potential to be a big step toward turning RL fine-tuning into a more predictable engineering discipline and less of a “guesstimate”. That could save an absurd amount of time and money, letting us develop increasingly capable models in a smarter way.

Here’s the link to the paper.