Why Are People Surprised That AI Gets Worse With Longer Instructions?
Basic information theory explains why AI loses accuracy over long chains of instructions: without error correction, errors accumulate at each step, for humans and LLMs alike.
This article is more than a year old. Some of its findings may have changed since.
It's basic information theory: errors accumulate without error correction
Why are people surprised that AI gets worse when following longer instructions?
We've been seeing headlines like:
- "Advanced AI suffers 'complete accuracy collapse' in face of complex problems, study finds" (The Guardian)
- "Why Superintelligent AI Isn't Taking Over Anytime Soon" (The Wall Street Journal)
- "Why We're Unlikely to Get Artificial General Intelligence Anytime Soon" (The NYTimes)
"AI systems become less accurate as they work through multiple steps of problem solving." Of course they do!
Information Theory 101
Since most people don't get much exposure to information theory in school, let me share a few principles (experts, bear with me):
1) Solvable problems have a theoretical complexity: they require a certain amount of resources (time, memory, etc.).
2) Solving problems requires going through step-by-step instructions (an algorithm), which crucially requires error correction.
What happens when you don't have error correction (which applies to both human brains and LLMs)? No matter how simple the algorithm, you become less accurate with each step toward the solution. This is exactly why the Soviets used teams to double-check basic arithmetic calculations before computers existed!
The Math is Straightforward
If you follow any set of instructions with a small probability of error at each step, those errors accumulate quickly — often exponentially — especially in tightly coupled reasoning chains where steps depend on each other.
This is basic information theory — no mystery here.
What Are Research Papers Actually Measuring?
So what are research papers like Apple's "The Illusion of Thinking" actually measuring? They're not testing how well AI can follow instructions for complex problems. They're measuring how well human brains versus AI can find shortcuts to solutions.
And yes, this varies based on what each has been exposed to during training - again, no surprise there. But the core limitation remains: neither humans nor AI can reliably follow long instructions without external support.
Promising Approaches Don't Eliminate Physics
AI researchers are proposing "better" approaches like JEPA (which I use in my own research). These are promising for building better internal representations, but they don't eliminate the need for error correction. They might help AI get better at skipping steps, but they can't bend the laws of physics.
The Only Reliable Solution
The only way to reliably solve problems, complex or not, is to:
3. Figure out the algorithm
4. Have access to error correction to follow that algorithm without compounding errors (a few Soviets with paper and pencil can work)
Let's make this crystal clear for anyone new to computer science or physics: humans, LLMs, or any other neural network all need access to error correction to follow long sets of instructions. That's it.
The same researchers who showed AI collapse on long instructions observed that when you give LLMs access to error-correction tools, they perform much better. Shocking, right?