LLMs Are a Transitional Architecture

The first thing large language models impress on you is the waste. Billions of matrix operations for a single token. Then just as many for the next one. The KV cache spares you from recomputing everything. Mixture-of-experts architectures activate only a fraction of the weights. The order of magnitude does not budge. Against a regime like that, complexity notation stops telling you much.

The observation is correct. It does not lead where people assume. The decisive limit of LLMs is not their cost. It is their rigidity.

The inefficiency case is aimed at a moving target

The efficiency argument ages badly because it conflates two distinct trends.

At constant quality, the unit cost of a token falls year after year. Andreessen Horowitz named the phenomenon LLMflation back in November 2024: a tenfold drop annually, which works out to a factor of a thousand over three years at GPT-3 level quality. The slope is gentler at GPT-4 level, where there is less history to go on, but the direction is not in dispute. Distillation, quantization, sparse activation and purpose-built silicon are all doing their work, and open-weight models have turned that good enough at a fraction of the price into a commercial weapon.

What is exploding is not the unit cost. It is usage volume, and the price of frontier training runs. Conflating the two distorts the public debate.

The comparison with the brain calls for the same caution. We like to set the model’s brute force against the elegance of living systems. But a hundred trillion synapses firing in parallel is brute force too. The difference comes down to a single number: roughly twenty watts. The honest argument is about energy efficiency, not about the nature of the computation.

And the brain does not start from nothing. It inherits an architecture selected over hundreds of millions of years, a form of pre-training considerably more expensive than a few months of GPU time. It then needs some fifteen years of data before it becomes useful. Comparing a model fresh out of a training run to an adult cortex is comparing a finished product to an organism that has never stopped learning.

The real weakness: frozen weights

Here is what actually separates the two systems.

The brain is a machine for continuous learning. It tries something, observes the result, compares it against everything that came before, generalizes, physically reinforces the pathways it uses most, and prunes what is outdated or wrong. In a loop, without interruption, for a lifetime.

The LLM comes out of training with its weights frozen. It does not optimize itself through use. Whatever you teach it in a conversation vanishes when the conversation ends. Fine-tuning and external memory, like the MEMORY.md file that Claude Code maintains on its own, work around the problem. They do not solve it. The distinction matters: for a great many uses, a persistent working context is entirely sufficient, and the illusion of learning it provides works perfectly well in practice. But a memory is data re-read on every request, not an acquired skill. The model is better informed. It has not become better. A coding agent reopens your repository every morning as though it had never seen it. And the moment you seriously alter the weights of a trained model, you degrade what it already knew. This is catastrophic forgetting. Continuous learning remains an open problem.

A system that cannot learn from its own use is not a final architecture. By construction, it is an intermediate step.

Generating token by token does not preclude planning

It is tempting to conclude that the answer is simply improvised as it goes, with no structure and no intent. That is inaccurate, and saying it weakens the critique.

Interpretability work over the past few years shows that the model’s internal state, before the first token is even emitted, already encodes a significant part of what follows. In Tracing the Thoughts of a Large Language Model (Anthropic, March 2025), the researchers set out to demonstrate the opposite and found their hypothesis overturned: on constrained tasks such as rhyming, you can observe circuits selecting the target word several words ahead, then building the line to land on it. Decoding is sequential. Planning is not entirely so.

None of this means the model understands in the sense we mean. A refusal, an apparent conviction, are still directions in a vector space that a software scalpel can excise. What it does mean is that the image of a parrot drawing the next word from a hat is obsolete. Leaning on it leaves the argument open to an easy rebuttal.

A return to symbolic AI will not be enough

The natural instinct, coming from software engineering, is to replace those probabilities with a structured, shared knowledge base. A system that would actually know what it is handling.

That road has already been traveled, and thoroughly. It is symbolic AI, expert systems, the Cyc project and its decades of encoding human knowledge by hand. It lost to statistical methods, and not by accident. Richard Sutton drew an unwelcome but durable lesson from it, The Bitter Lesson: in every generation, the approach that exploits available compute ends up beating the one that encodes expertise manually.

The lesson has just repeated itself in front of us, as it happens. Reasoning models, from o1 to DeepSeek-R1, no longer spend compute only during pre-training but at answer time, exploring several lines of attack before committing, and that internal deliberation was learned through reinforcement, not prescribed. It is the Bitter Lesson a second time, now applied to inference.

There is an irony in the fact that the largest shared knowledge base ever built, Wikipedia, is losing its human readers at the very moment it stands as the most frequently cited source of the models it helped train. Anyone proposing a return to the knowledge base today has to answer that argument. Otherwise the proposal gets filed immediately under: already tried, already failed.

The real dividing line is not statistical versus symbolic. It is verifiability.

Where the instinct deserves a second look is on how the probabilistic and the deterministic fit together. Just not in the way people picture it.

The usual picture is a hybrid in which the LLM talks to the human, formalizes a request, and hands it to an engine that does know what it is doing. The trouble is that this engine does not exist. Producing a non-trivial program from a specification has been the dream of program synthesis and formal methods for fifty years. It has never reached industrial scale. The LLM is precisely what was missing to bridge the gap between specification and implementation.

So the hybrid already exists, but running the other way. The deterministic part is not downstream. It sits inside the loop, and it generates nothing: it verifies. The model proposes. The compiler, the type checker, the test suite and the formal prover decide. The model tries again. That is what today’s coding agents do. It is also what allowed AlphaProof, paired with the Lean proof checker, to reach silver-medal standard at the International Mathematical Olympiad.

Hence the reversal. Code is not the domain where LLMs are weakest. It is the one where they are most reliable, because it is the only one where the output can be verified mechanically and immediately. A hallucination in code costs you a compile error or a red test. A hallucination in a briefing note or a piece of legal advice goes unnoticed and propagates. It is also why developers are the first profession hit: not because code is easy, but because code can be checked.

So the right question is not “LLM or structured system.” It is: who verifies?

Where a mechanical verifier exists, the probabilistic becomes usable, because its errors get caught. Where none exists, you are left with probabilities and no safety net. That is where the risk concentrates. And the verifier is not only the machine: when nobody reads the code they deploy any more, the net may still be there, but you have taken it down yourself.

On-device models do not need retraining every week

That leaves deployment. Small local models on phones make sense: latency, cost, privacy, data sovereignty. They are arriving. Apple stores twenty billion parameters in flash memory and activates only a fraction of them per request. For anyone who wants to keep their data at home, local inference has become a serious option, even paying for it three times over.

Retraining those models frequently to keep up with the news, on the other hand, is a dead end. Current events do not get written into the weights. They get retrieved at query time, through search or document indexing. The on-device model is meant to stay small and stable, and to reason over fresh external sources. Separating the reasoning engine from the stock of facts is, besides, the only way to make the whole thing auditable. That is no small matter when sovereignty is on the table.

What this allows, and what it does not

LLMs are not a dead end, and the inefficiency case will age badly. But they are systems frozen by construction, incapable of learning from their own use, and entirely dependent on an external verifier for their reliability.

That is enough to transform the software industry. It is not enough to be the final architecture. A transition toward what, then? Toward a system that keeps the verifier in the loop, that spends its compute searching rather than storing, that reasons over representations of the world rather than sequences of words, and above all one that learns from its own mistakes instead of repeating them until the next training run. The candidates exist, world models in LeCun’s work, continuous learning elsewhere. But nobody today knows how to assemble the next step.