ZeroSearch, the Search Simulator for LLMs
Published on
I don’t know if you know this, but with the arrival of LLMs with web search access, AI labs started running into a few problems while training these models to search better with Reinforcement Learning.
The first is the HIGH spending on Google search APIs during training (imagine the number of real searches that get made until you find the model’s sweet spot). Another is the fact that the quality of the results coming from the web can be a lottery: sometimes great, sometimes a bunch of stuff that has nothing to do with anything, low-quality information, or even contradictory. That messes up the LLM’s learning, because it’s trying to learn from “informational garbage”.
Well, it looks like the folks at Tongyi Lab (from Alibaba Group) found a REALLY interesting way to solve this: ZeroSearch.
The trick is great: instead of making the model knock on Google’s door via API all the time during Reinforcement Learning training, they use another LLM to SIMULATE those searches.
It works like this:
1 - They take an LLM (or SLM, since it can even be a smaller one) and do a light fine-tuning on it with the goal of turning it into a “search API simulator”, capable of generating both useful documents and noisy documents (with irrelevant or incorrect information) for a given query.
2 - During the training of the main LLM, they use a “school curriculum” strategy, meaning they don’t throw all the “bad” documents at it at once. At the start of training, the search simulator generates more “good” documents that are easy to understand. As the main LLM learns, the simulator starts to “make things harder”, generating a bigger proportion of noisy documents. This forces the main LLM to get better and better at reasoning, at separating the wheat from the chaff, and at really understanding which information is useful to answer the original question.
The advantages are that, since we’re not hitting Google’s API on every query and are only simulating those searches, the API cost drops to zero (hence the name “ZeroSearch”). On top of that, researchers get much more control over the quality of the documents the main LLM receives during training, being able to adjust the difficulty of the simulated search task, without depending on the Russian roulette of real search results.
The results showed that using a 7B-parameter model as the simulator already achieves performance comparable to using the real Google search API in the process. And more: when they used a 14B model as the simulator, they even managed to BEAT the results obtained by using real Google search in training (of course, within their experimental setup, but still, that’s a hell of an achievement!).
I found the idea super elegant. It’s the kind of solution where, once you see it, you think: “how did I not think of that before?”.
Here’s the link to the paper.
