The Illusion of Reasoning in AIs

Published on

This week, by far, the AI paper that made the most noise was one from Apple (who would have thought) called “The Illusion of Thinking”.

We always talk about “LLMs”, but there’s a new acronym for language models that “reason” for a while (using Chain-of-Thought) before producing the answer: LRM (Large Reasoning Model). These models (like o3, Gemini 2.5 Pro, DeepSeek-R1) have been performing much better on most benchmarks than “normal” models, which start answering right away.

The problem the paper points out is that evaluations of these models focus only on the final answer in math/code benchmarks, which can suffer from data contamination (something we always point out on IA Sob Controle), and which don’t show what happens in the reasoning process itself, so we can’t tell whether the model really “reasoned” its way to the answer, or just “memorized” a pattern.

Their trick was to use famous puzzles where they could control the complexity (like Tower of Hanoi, River Crossing, etc.) to analyze not only the final answer, but the whole CoT “reasoning” process.

For these puzzles, they split the difficulty into 3 levels: easy, medium, and hard (like, the number of disks used). On the easy ones, the LLMs without reasoning performed better. On the medium ones, the LRMs were a lot better. On the hard ones, both collapsed.

The finding that caught my attention the most was the “accuracy collapse point”. For all the models tested, as the complexity of the problem went up (for example, by adding more disks to the Tower of Hanoi), at a certain point the models’ accuracy CRASHED, and not gradually, but straight down to zero.

On top of that, as they got close to that point, the LRMs started using FEWER tokens for the “reasoning” process, even though they could use more. It’s as if they “gave up” earlier.

Even when they were handed the exact algorithm for the solution, performance didn’t improve much, and the accuracy collapse happened at practically the same level of complexity. In other words, the limitation isn’t only in figuring out the solution, but in executing the steps consistently and precisely. So, they don’t really think.

People who were already AI skeptics loved it. Gary Marcus called it a “bucket of cold water” for the AGI optimists.

Plenty of others, like Yann LeCun, said “Ok, but what’s new?”. LeCun, in fact, has been saying for a long time that LLMs won’t be the path to AGI, and that we’ll need other architecture(s). Marcus Mendes wrote a post about this, which is linked at the end of this one.

One criticism of the paper that I found interesting says it’s irrelevant, because it only evaluated whether LLMs can execute a complex formal algorithm without having a memory to store the state, and that they’re bad at that, just like we humans are too; and that if these models had access to a notepad or a Python REPL, they would destroy all of these tests.

  • Link to the Apple paper
  • Link to Marcus Mendes’s post