SkillsBench: how you should use Skills in Claude Code
Published on
Everybody talks about increasingly autonomous AI agents, but can they already create their own knowledge to solve complex problems on their own?
A super interesting paper just came out, made by a huge global collaboration (Stanford, Amazon, Chinese universities, etc.) called “SkillsBench”, and it set out to investigate exactly that. The researchers built a big benchmark, with more than 80 tasks across 11 different areas, to understand how these agents perform with “Skills” (which are basically structured packages of procedural knowledge, like step-by-step guides, best practices and templates).

They ran more than 7 thousand trajectories with the leading models on the market (Claude Opus 4.5, Gemini 3 Pro, GPT-5.2, etc.), and found that when agents are given curated Skills (made by humans), their success rate goes up by 16.2% on average!
And of course, that impact varies a lot depending on the domain. In healthcare tasks, which demand super specific knowledge, the jump was an absurd +51.9%; in software engineering, the gain was +4.5% (probably because a lot of that coding reasoning is already in the models’ pretraining).
But what I found most surprising was the “self-generated Skills” test. When the researchers asked the models to generate their own plan and step-by-step before solving the task… the benefit was basically ZERO. In other words, today’s AIs still can’t reliably create the procedural knowledge they themselves need to succeed at a task.
Another really cool finding is that smaller models armed with good Skills (like Claude Haiku) can beat much bigger and more expensive models (like Claude Opus) trying to solve the same problem from scratch, without those instructions.
One more point worth mentioning: according to the researchers, less is more. In other words, giving the model 2 or 3 focused, modular skills got the best performance, while extremely long and complete documentation ended up confusing the models and hurt their performance.
Giving my opinion, this is one of the most practical findings we’ve had in a while about agents. We talk all the time on IA Sob Controle about the future belonging to agents, but this paper shows that it’s not enough to just throw a giant (and expensive) model at a complex problem. And the fact that AI is still pretty bad at creating its own skills shows that our “human-in-the-loop” role is still safe for a while.
I know that at Quinto Andar folks have been using skills a lot in their AI agents. What about at your company? If so, which ones and for what?
Here’s the link to the paper!