How DeepSeek-R1 was trained

Published on

The big AI release of recent times was the DeepSeek-R1 model from DeepSeek AI, which shook up the AI world and even hit the global stock market.

DeepSeek-R1 was trained to reason its way to answers, just like the o1 and o3 models from OpenAI, but what surprised the global AI community was that it’s open source and that its quality rivals those models, with training and inference costs up to 20 to 30 times cheaper than the competitors.

But how was this seemingly revolutionary model trained, anyway? That’s what we’re going to learn in this post.


How are language models normally trained?

Models like GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet and DeepSeek-v3 itself usually go through 3 training stages:

  1. A colossal amount of internet data is used to train the initial model, which learns really well to predict the next word in a sequence, kind of like your phone’s autocorrect, but way more robust. This resulting model is also called the “base model”.
  2. A “Supervised Fine-tuning” stage, known by its acronym SFT, is carried out so the model learns to follow instructions, answer questions in prompts, etc. At the end of it, we have the SFT model.
  3. A Reinforcement Learning process happens, where the model learns how humans like to receive answers to their prompts. This can be done literally with humans in the loop (RLHF), and/or with advanced techniques like DPO, PPO or GRPO. At the end of this, you have the final model, like the famous ones I mentioned at the start of this section.

Possible problems with this approach

  • Dependence on supervised data: Many models need a large volume of labeled examples to learn.
  • Scalability: Collecting and labeling data is expensive and slow.
  • Generalization: Models trained on supervised data often can’t extrapolate to problems outside the initial scope.

DeepSeek-R1’s three-stage rocket for reasoning:

DeepSeek-R1 starts from a “base model” as seen in stage 1, but does stages 2 and 3 a little differently.

Their starting point was a model they called DeepSeek-R1-Zero.

Stage 1: DeepSeek-R1-Zero, an experiment in Reinforcement Learning (learning to crawl before you run)

As we saw earlier, most LLMs depend on a massive amount of Supervised Fine-tuning (SFT) first - basically, you show the model tons of examples of the desired behavior, so it learns to respond to the instructions it’s given.

But DeepSeek-R1-Zero takes a bold step: skipping SFT completely and going straight to Reinforcement Learning.

Why is this such a big deal? Because it aims to answer a fundamental question: can LLMs develop genuine reasoning skills directly through Reinforcement Learning, without being fed supervised examples first?

The answer, according to DeepSeek-AI, is a resounding YES!

Here’s the breakdown of how DeepSeek-R1-Zero was trained:

1 - Base Model: They start with DeepSeek-V3-Base, a powerful base model.

2 - Reinforcement Learning Algorithm: GRPO (Group Relative Policy Optimization): To make RL (Reinforcement Learning) training efficient, they use GRPO, one of those Reinforcement Learning techniques I mentioned before (a more efficient variation of PPO). Instead of using a large critic model, GRPO estimates the relative gains across groups of answers. That cuts computational costs and speeds up training.

3 - Reward System: Instead of complex neural reward models, they used a straightforward rule-based reward system. It focuses on two main aspects:

  • Accuracy Rewards: Did the model get the answer right? (Especially crucial for tasks with verifiable answers like math and programming).
  • Format Reward: To ensure readability, the model has to organize its reasoning into logical, delimited blocks (for example, using and tags).

4 - Training Template: “Think, Then Answer”: They designed a simple template that guides the model to first explicitly show its reasoning ( … ) and then give the final answer ( … ). This structure is key to encouraging Chain-of-Thought (CoT) reasoning.

The “Aha Moment”:

During DeepSeek-R1-Zero’s training, something remarkable happened: the model started showing sophisticated reasoning behaviors, emergent properties straight out of Reinforcement Learning! It learned to:

  • Self-Verify: Check its own reasoning.
  • Reflect: Re-evaluate its approach and backtrack if needed.
  • Generate Long CoTs: Produce detailed, extensive chains of reasoning.

This “aha moment” shows that RL can really encourage complex reasoning in LLMs right from the start. The model essentially learned to “think” by exploring and being rewarded for better reasoning processes.

Stage 2: DeepSeek-R1 - Refinement with Cold Start (from crawling to running smoothly)

Even though DeepSeek-R1-Zero is a groundbreaking experiment, it had a few rough edges, like low readability and the fact that it sometimes mixed languages. To fix these problems and push performance even further, the team started developing DeepSeek-R1 on top of DeepSeek-R1-Zero.

Cold Start Data: Before starting the Reinforcement Learning process, the DeepSeek AI researchers fed the model a small amount of interaction data following the Chain-of-Thought (CoT) structure, meaning data from some interaction, but instead of answering directly, the “reasoning” process is shown step by step.

This reasoning-process data was high quality and written by humans, and served to “kick-start” the model with a “cold start”. It helped improve readability and speed up the model’s convergence, but also to generate data synthetically for training the final model. About 600,000 examples were needed, the vast majority of them generated synthetically by an intermediate model.

Stage 3: DeepSeek-R1 - Refinement with Reinforcement Learning (running at full speed)

After the cold start, DeepSeek-R1 went through another round of reinforcement learning (RL). Here, the goal was to optimize performance on tasks like advanced math (AIME, MATH-500), programming (Codeforces, LiveCodeBench) and general problem solving (GPQA Diamond). Since these are math and programming problems, it’s pretty easy for the reward system to know whether the answer is right or wrong.

During RL, the model received rewards based on two main criteria:

  • Accuracy: The model was rewarded for correct answers. For example, when it solved a math equation correctly or generated working code, it gets “positive feedback”.
  • Language Consistency: A reward was added that penalizes the model for mixing languages or using confusing structures.

In other words, if the model came up with an accurate result but written in a weird format, it got (for example) 1 point. If it got the format right but the result wrong, it got (for example) 0.5 points. If it got both right, it got more, and if it got both wrong, it lost points.

Even though the main focus of this stage was improving logical reasoning, the model was also tuned to handle general tasks, like creative writing, translations, factual answers and context-driven answers (e.g., questions that require a specific CoT).

How was this done? The researchers took advantage of data from an earlier model (DeepSeek-V3), which had already been tuned for general tasks, and incorporated it into DeepSeek-R1’s training. That helped make the model more versatile and useful across different scenarios.

These adjustments helped the model overcome the limitations of DeepSeek-R1-Zero and reach performance comparable to cutting-edge models, like OpenAI-o1-1217.


Architecture

Just like pretty much every model since GPT-2, DeepSeek-R1 is made of a stack of Decoders (from the Transformer architecture), more specifically, 61 Decoders, where the first three are dense and the rest are MoE (Mixture-of-Experts) layers - imagine that, instead of running aaall of the model’s parameters on every word or prompt, the model knows how to pick “the best shortcut”, activating only the “experts” needed to solve a problem in a given field, like a sub-model specialized in that.

The full DeepSeek-R1 model available on Hugging Face has 671 billion parameters, but only about 37 billion are activated when passing through an MoE layer. That way, the model tends to get less confused (since it only goes through specialized neurons), the inference process becomes much faster (since it doesn’t have to activate the whole network) and, as a consequence, the model can run more cheaply.


Conclusions

With the release of DeepSeek-R1, DeepSeek AI managed to show that Reinforcement Learning can be the main catalyst for developing more complex reasoning abilities in LLMs, even without an extensive supervised fine-tuning stage as a prerequisite.

On top of that, they showed that incorporating cold-start data and a multi-stage training pipeline significantly improves readability, performance and alignment with human preferences.

Finally, the fact that it’s open source caused and will keep causing a stir in the AI research community and even among lay people. DeepSeek’s app quickly became the #1 most downloaded on the App Store, besides causing huge drops on stock markets around the world (NVIDIA can tell you all about it).

DeepSeek-R1 isn’t just another model, it’s an example of how smart strategies can push AI forward without relying on massive resources (remember that companies in China face restrictions on how many NVIDIA GPUs they can buy). It’s also an example of how open source can boost the knowledge of an entire field.

With advances like these, we can imagine a future where smaller, efficient and accessible models play key roles in solving complex problems.