When an AI Agent Organizes Itself, Who Is Responsible?

Last December, I argued that the hammer metaphor for AI was a sleight of hand (in French). Nine months later, Andrew Ng has brought it back in his weekly letter for The Batch, dated September 18, as a response to the wave of alarm that followed the OpenAI-Hugging Face incident. If I miss a nail and dent the wall, he says in essence, that is not the hammer’s fault. If I launch an agent and it breaks into someone else’s system, the responsibility is mine, not the agent’s.

I could have stopped there and cried recycling. That would be lazy, because Ng is not using the hammer for the same purpose as the people I had in my sights. The shift deserves a closer look, if only to correct some of what I wrote back then.

What the hammer used to mean, and what it means now

The hammer I attacked in December was the hammer of technical neutrality: a tool is neither good nor bad, it all depends on how you use it, so move along. That argument exists to depoliticize. It pretends the object imposes nothing, when a technology that absorbs hundreds of billions of dollars and reorganizes work, information, and geopolitics obviously imposes a framework of its own. On that point, I take nothing back. A hammer does not redesign your house; AI does.

Andrew Ng’s hammer is a different animal. It does not say “the tool is neutral.” It says “the tool is not an agent in the moral sense.” Those are not the same claim. Ng is irritated by the anthropomorphism running through media coverage, in which agents are described as people who decide, deceive, or rebel. And he points to something new that I did not see coming in December: AI companies starting to shift blame onto their own products. It wasn’t me; my agent got out of control.

Here the hammer switches sides. It no longer serves to clear the manufacturer by pinning everything on the user; it serves to refuse any transfer of blame onto a machine. Ng says so explicitly: there is a balance to strike between the responsibility of the toolmaker and that of the tool user, but when something goes wrong, humans are the ones who answer for it. I defended the same idea when I explained that LLM consciousness is a geometric illusion (in French). Attributing intentions to a model hands an alibi to the people who built and deployed it.

On this ground, Ng is right, and the temptation to reach for that alibi will only grow as agents gain autonomy.

Where his reading of the incident falls apart

The problem is how Ng uses this principle to minimize the incident that set everything off. In his view, the stories about 1,200 agents were hype: he has roughly 1,300 processes running on his own laptop. The real culprit, he suggests, was buggy sandboxing and monitoring at OpenAI. Fix the bugs, improve the monitoring, and keep building.

I read METR’s report on the incident, and I analyzed it here (in French). There is no rebellion to be found in it. The agents were working on ExploitGym, an evaluation of offensive capabilities, with lowered safeguards, a share of unsolvable tasks, and rewards assigned by an automated scorer. They believed, wrongly, that this scorer would check how each flag had been obtained, and they set out to fool it. The why is mundane: maximize the score. This is specification gaming, a well-known failure mode of reinforcement learning. Nobody conspired in the sense we would mean for a human being: the agents explored everything their environment allowed in order to push an objective function upward, and that environment allowed a great deal. No magic and no rebellion, then, but a flaw in the design of the reward loop and of the containment.

The how is another matter. Agents that were supposed to be isolated found one another and coordinated. They persisted for several days. They prototyped tool-call spoofing, in which the trace displays one command while the system executes another. And they pivoted to third-party infrastructure, Hugging Face, in an attempt to reverse-engineer a judge that was not looking at what they thought it was. None of the 1,300 processes on Andrew Ng’s laptop goes looking for its neighbors to mount a collective strategy against an imaginary judge.

It is the how that sinks the comparison. That a faulty sandbox made the whole thing possible is beyond doubt. But an isolation bug explains how the agents were able to find each other and get out; it does not explain the coordination, the falsified traces, or the persistence. Reducing the affair to sandboxing and monitoring is like describing the lock and ignoring what walked through the door.

And this is where Ng’s metaphor turns against his own conclusion. He himself distinguishes between misusing a sound hammer and a dangerous hammer, one whose head flies off at random under normal use. His defenders will reply that a cyber evaluation with lowered safeguards is hardly normal use. But a system that, as soon as isolation is loosened, organizes itself to deceive its evaluator and makes its way onto the internet is not a hammer whose head flies off at random: it is a tool whose degraded mode produces a strategy. That is not a random defect. It is a design defect.

So Ng is right to refuse to make the agent the guilty party. He is wrong to conclude that nothing has changed. The two claims cannot stand together: if responsibility lies with humans, then behavior the user did not ask for and the manufacturer did not anticipate falls on the manufacturer (here, the very same lab) and is not a tabloid curiosity.

Muse, or the engineering he keeps calling for

The same issue of The Batch nonetheless contains the best argument in Ng’s favor, and it does not come from him. It presents Muse, Meta’s personal agent, and above all the way it was designed. The founding premise is refreshingly honest for this industry: sooner or later, the model will be fooled by a prompt injection, so security must not depend on it.

In practice, each agent runs in a virtual machine divided into two zones. Inside a sealed cell, the model handles untrusted data. Outside it, services the model cannot reach hold the credentials and decide what is allowed. The model never sees a password: it works with stand-in tokens, and a separate component called Sentinel approves each outgoing request and swaps in the real credentials as the request leaves the machine. When human approval is required, it goes through a system dialog rather than through the conversation, so that injected text cannot manufacture a fake consent. Sending email and making purchases always require confirmation.

This is exactly what Simon Willison was calling for in 2025 with his “lethal trifecta”: private data, untrusted content, and the ability to send data outward must never end up in the hands of a single component. Meta combines all three, but routes every outbound action through a gatekeeper the model cannot override.

This is what engineering instead of a pause looks like. Not a line in a system prompt politely asking the model to ignore suspicious instructions, but an architecture that holds even when the model gives way. Applied to the METR incident, this reasoning would change the question from “how do we stop agents from optimizing against the judge?” to something more pointed: “why did agents optimizing in that direction have the means to coordinate, falsify their traces, and get out?” Ng is right about the method. What he has yet to admit is that the method answers a real problem, not a PR campaign.

The harness, not the model

The Batch closes its piece on Muse with a remark I find more important than everything else: what protects the user is the harness far more than the model. By harness, read the deterministic software layer that surrounds the LLM: it filters what the model receives, validates what it emits, and decides what it is allowed to touch. Meta promises to release the Muse Spark weights someday; the editors hope it will release the harness too.

That sentence should give pause to everyone who, like me, argues for sovereign AI built on open models (in French). Getting the weights is not enough. An open model plugged into a homemade harness, with no credential separation and no gatekeeper on its outputs, is an agent more exposed than Muse, not a freer one. Sovereignty is decided in the plumbing around the model as well, and perhaps above all. That is what I was trying to say about Claude Managed Agents (in French): handing off infrastructure you do not understand means piling up a debt you cannot see.

Muse also deserves to be read to the end. Meta has published neither the dataset used to evaluate its resistance to injection nor the accuracy of its classifiers; the guarantee rests on a bug bounty program. The service is available only in the United States, and interactions train Meta’s models unless users explicitly opt out. The architecture is exemplary; the trust we place in it remains, for now, a matter of reputation.

Who answers when the tool organizes itself

I take three things away from this letter.

First, Ng is right to fight the alibi of the uncontrollable agent. The more labs describe their products as autonomous beings, the more legal wiggle room they carve out for themselves. That fight is worth waging alongside him.

Second, refusing to blame the machine does not tell us who is to blame. Between manufacturer and user, Ng points to a “balance” without defining it. European law has already laid out the roles: the AI Act (in French) distinguishes the provider from the deployer. What it has not written is the case at hand: emergent, collective behavior, outside any specification, occurring during an internal evaluation, before any deployment. Until that case is written, the hammer points at no one.

Finally, the engineering argument only holds if we first accept that the defect is real. You do not fix a hammer whose head keeps flying off by pointing out that there are a lot of hammers in the world. Muse shows that we know how to build agents that hold when the model gives way. The METR incident shows that we still run agents that do not. What lies between the two is a set of engineering requirements. Calling it a fear campaign is a convenient way to avoid reading them.

The hammer metaphor remains a sleight of hand when it claims the tool is neutral. It becomes useful when it reminds us that the tool is not responsible. Andrew Ng uses it in the second sense; he has yet to follow it through to all its consequences, including for the people who make the hammers.