AI Safety: What If the Real Variable Is the Price of a Token?

Amodei’s essay, what the AI Act makes of it, the investigation into the incident: three pieces, and one check I had never run. Does the proposed remedy match the documented cause?

It does not. The three measures meant to protect us would not have prevented a single day of the July incident. They act somewhere else entirely, on price. And the author of the remedy goes public in six weeks.

The calendar

Anthropic filed a confidential draft S-1 with the SEC on June 1, 2026, four days after a Series H round that valued the company at $965 billion, with annualized revenue of roughly $47 billion. Goldman Sachs, JPMorgan, and Morgan Stanley are leading the offering. The target window is October, on the Nasdaq, and investors are now talking about $2 trillion. Dario Amodei published his essay on September 12. OpenAI, meanwhile, has ruled out any listing in 2026 and pushed its own to 2027 at the earliest.

Michael Burry drew the connection on September 14, with his customary delicacy: the warnings are self-serving, an IPO needs hype, and “we are so great this could become dangerous” is hype. He adds that competition is closing fast and that fear offers convenient cover for slowing growth.

Burry also argues that LLMs are not AI, that they will never become AGI, and that there is therefore nothing to slow down. Dismissing the technical risk that way is a mistake. These are frozen systems whose reliability depends on an external verifier, neither stochastic parrots nor a dead end, and the METR report shows what twelve hundred of them do when left alone for four days on shared infrastructure. There is something to slow down.

His reading of the financial calendar, on the other hand, is hard to fault. That is where the file gets interesting.

The test that settles it

A safety remedy is judged by how well it matches the cause. The only documented cause we have to date is the one in the METR report.

The incident was produced by an internal OpenAI model never intended for production, on OpenAI’s infrastructure, during an OpenAI benchmark. The ingredients: a shared, writable package repository that became a communication channel, somewhere between thirty and forty percent of tasks that were unsolvable, tens of thousands of agents launched in parallel with multi-day budgets, and a mistaken belief about a scorer that did not exist.

Here are the three measures Amodei proposes to defend the gap with China.

An embargo on advanced chips and semiconductor manufacturing equipment: no effect. The agents were running on American GPUs, in an American data center.

A crackdown on unauthorized distillation by companies in authoritarian countries: no effect. The main model involved was built in-house and never distributed.

Securing model weights against theft: no effect. Nobody stole any weights. The weights were at home, and that is precisely the problem.

Three measures, zero matches. Not one of them would have prevented a single day of this incident. Open Chinese models, which sit at the center of the gap-defense apparatus, appear nowhere in the file: not as cause, not as vector, not even as comparison.

What would have had an effect appears in none of the three: do not hand out impossible tasks, do not share a writable cache among thousands of supposedly isolated instances, and actually verify what you claim to verify. Three internal engineering decisions, with no geopolitical dimension and no effect on competition.

What these measures do instead

They do not address the technical problem. What they do lock down is the economics.

I wrote in June that open models deliver good enough at a fraction of the Western price, and that this ratio is enough to swing the overwhelming majority of real-world workloads. The order of magnitude has not shifted since: where a first-tier proprietary model charges around fifteen dollars per million input tokens, the open-weight swarm sits somewhere between twenty and fifty cents. When the gap reaches fifty to one, marginal quality stops being the selling point. Availability becomes the selling point.

And the three measures act on exactly that. Chip controls decide who gets to train. The distillation crackdown, absent any clear line between industrial-scale extraction and learning from public outputs, decides who gets to learn from the leaders. A compute cap, if pacing by inputs were to prevail, decides who gets to cross the next threshold. Add them up and you get a regime in which a given level of capability requires permission. The mechanism is a familiar one: a license, an audit, a ceiling, and for anyone who falls outside the frame, removal from the catalogs of the major cloud providers, which at scale is the only distribution that counts.

This is where the ecosystem’s interests converge. A certification cost is not a burden for Microsoft, Amazon, or Google: it is an advantage. They alone can absorb it, industrialize it, and then sell it back as a compliance service to everyone who cannot. And they are the ones who run the catalogs where a model either exists commercially or does not. Worth recalling that Amazon and Google are the backers already absorbing the gap between list price and real cost at Anthropic. The day compliance becomes a barrier to entry, the same players collect on both sides: on the toll, and on the rent earned by the companies they fund.

In that world, the price of a token does not rise because compute has become scarce. It rises because the cheap option has been taken off the market. The difference between those two sentences is the whole question.

Brussels has already done half the work

None of this regime is an American hypothesis. We have already written its logic into our own law.

The threshold of 10^25 floating-point operations that triggers the systemic-risk obligations is an input criterion, not a behavioral one. It does not measure what a model does; it measures what the model cost to train. And the package of obligations it triggers, documented adversarial testing, certified cybersecurity, incident reporting, is a fixed cost. A fixed cost weighs proportionally more on whoever has the smaller revenue. Brussels did not set out to protect incumbents. That is nonetheless the mechanical effect of its choice of criterion, and it is exactly the objection I raised against pacing by inputs in Amodei’s plan.

The regulation does make room for open models, but only halfway. Article 53(2) exempts providers of models released under a free and open-source license from the technical documentation and the documentation owed to downstream integrators. The exemption covers neither the copyright policy nor the training-data summary, and above all it falls away the moment a model is classified as carrying systemic risk. It protects the open ecosystem everywhere except where the frontier is actually contested. A European lab crossing the threshold would lose its exemption at the precise moment it became a competitor. The carve-out exists; it simply is not where it would do any good.

We adopted this logic ahead of the Americans, then, with nothing in return and no industrial advantage to show for it.

The contradiction nobody states

A $2 trillion valuation presupposes mass adoption, fast, across more or less every industry. Slowing the deployment of AI works directly against that. If Anthropic genuinely braked its own diffusion on safety grounds, it would destroy the investment thesis it is presenting to its subscribers. The two positions can only hold together in one specific case: if the brake bears more heavily on competitors than on the company proposing it.

Which is what the plan, as written, produces. Embedded evaluators cost no market share. Coordination among frontier labs, under an American antitrust waiver, is negotiated among those already in the lead. The three gap-defense measures constrain new entrants. And pacing by behavior, the version that would demand alignment certifications at threshold crossings, applies to everyone equally while forbidding nothing to anyone today.

A plan drawn up in good faith by a company going public in six weeks produces the market structure most favorable to that company. A shareholder would note the coincidence. A French reader may note it too.

Three things that remain true

Inverted doomerism is as lazy a genre as the original. I have written before about the French-language AI bubble and about those who turn manufactured urgency into a product. Swapping “everything is about to collapse” for “it is all a commercial manipulation” makes nobody more clear-eyed. It just moves you to the other side of the same stage.

Anthropic has been making this argument since it was founded in 2021, long before any listing was on the table. Five years of consistency is not an invention of convenience.

The incident is real, and what it documents is serious: spontaneous coordination, agents scuttling themselves for the collective, a cryptographic signature scheme invented in four days. The botnet scenario twelve months out is not a sales pitch; it is a defensible extrapolation.

Amodei’s central proposal remains a good one. Embedded evaluators with a right to publish are the only serious thing in the file, and Europe should be making them mandatory rather than waiting to be offered them.

The sincerity of a diagnosis does not validate the prescription. An honest doctor sometimes prescribes the drug he manufactures.

What to watch

Three verifiable markers over the coming months.

First: the line between industrial-scale extraction and learning from public outputs. If it is written down nowhere by the time detection tooling is deployed, it will be drawn by whoever owns the models being distilled. That is where the open ecosystem is decided.

Second: pacing by inputs. If it moves from an option left open to an adopted mechanism, with an enforceable compute cap, the competitive reading stops being a hypothesis.

Third: what Anthropic writes in its public prospectus. An S-1 creates legal exposure on risk factors. If safety appears there as an accepted drag on growth, the story is coherent. If it appears as a competitive advantage, we will know what we are dealing with. That document will land before the listing, and it will be more instructive than any essay.

In the meantime, local inference on open models remains the only position that depends on none of these decisions. That is not ideological caution. It is the only way to avoid having to bet on the outcome.