ARC-AGI 2 is out. Cognitive emergence goes on to chase that new objective.
Frontier models fall to 4% on ARC-AGI 2. The essay reads this low score as a sign that compositional problem-solving has begun to emerge in AI systems without explicit design.
This article is more than a year old. Some of its findings may have changed since.
Understanding cognitive emergence in AI systems and the surprising reality of compositional reasoning
Frontier AI models scoring 76-87% on ARC-AGI 1 fall to just 4% on version 2, far below the human average of 60%. Critics will rush to interpret this as proof that we're on the wrong track. They're likely missing the point entirely.
The Emergence of Compositional Problem-Solving
The 4% success rate of models with test-time adaptation reveals that genuine compositional problem-solving capabilities have begun to emerge in such systems, without explicit design. To put this into perspective: compositionality was deemed impossible by symbolic AI proponents.
Moreover, the "coherence time" of frontier models — the maximum number of tasks that can be completed autonomously with 50% reliability — currently doubles approximately every 7 months.
Physics Perspective on Cognitive Emergence
Some experts claim AI is fundamentally limited because training objectives don't take into account concepts like "understanding" or "reasoning." Centuries of physics offer a different perspective: the fundamental rules that create human intelligence also know nothing about these concepts. Cognitive capabilities emerge from lower-level rules in humans and machines alike.
Our brains do not manipulate perfect symbolic representations but rather noisy ones. The crisp symbolic experience we subjectively feel when solving ARC puzzles — recognizing a "hole" in a shape or the concept of "inside" — doesn't exist literally within our neural architecture.
It emerges as higher-level abstractions from noisy physical interactions.
Shared Cognitive Limitations
The fact that humans struggle with ARC-AGI 2 exposes the limits of our own compositional processing. Complex tasks requiring simultaneous tracking of multiple interacting rules push against the constraints of our cognitive architecture — a limitation we share with current AI systems, albeit at different scales.
To dismiss LLMs as "just distributions over tokens" is like dismissing humans as "just neurons firing." Both statements are technically accurate at one level of analysis yet profoundly misleading at another. As in physics, multiple layers of descriptions can coexist without contradiction. The statistical foundation of LLMs doesn't negate emergent cognitive behaviors any more than quantum mechanics negates molecular biology or our own conscious experience.
The Real Question
The real question isn't whether LLMs can develop compositional reasoning — ARC-AGI 2 suggests they already have. The question is when their compositional abilities will reach human-level thresholds.
ARC-AGI 2 heavily relies on visual compositional reasoning — currently not the strongest suit of frontier models primarily trained on textual data. If current trends in coherence time continue, however, that milestone may arrive sooner than many expect (within one to two years, according to the benchmark's creators).