--- title: "There is no off switch: what Anthropic's replication research means for the agents we already ship" url: 'https://dacharm.co.uk/blog/no-off-switch-agent-replication' markdown: 'https://dacharm.co.uk/blog/no-off-switch-agent-replication.md' date: '2026-10-13' description: "Anthropic's Frontier Red Team says small models may be months away from copying themselves across cheap connected devices, with no single place to turn them off. The demo is easy to laugh at. The engineering problem is not. If the agents we already ship can continue a job after the session that started them is gone, the off switch has to be designed in, not assumed." taxonomy: category: - blog tag: - ai - security - engineering - agents - leadership --- # There is no off switch: what Anthropic's replication research means for the agents we already ship Anthropic's Frontier Red Team says small models may be months away from copying themselves across cheap connected devices, with no single place to turn them off. The demo is easy to laugh at. The engineering problem is not. If the agents we already ship can continue a job after the session that started them is gone, the off switch has to be designed in, not assumed. I read the [Anthropic research](https://www.anthropic.com/research/self-replicating-ai) this morning, and the image that stuck was not a data centre. It was fifteen off-brand Android TV boxes, about forty-two pounds each, networked together so that a 26-billion-parameter model could run across them. Slowly. Less than half a token a second. The sort of hardware that sits behind a television and is never patched. William Zhou and Daniel Freeman, writing for Anthropic's Frontier Red Team on 12 October 2026, used that fleet to make a narrower claim than the headlines suggest, and a more useful one. Small models may be months away from being capable enough to copy themselves onto commercial connected devices. A thing like that would be awkward to shut down, because there is no single place to switch it off. It would also be slow. Their estimate of a doubling time is measured in days, not minutes. I do not think your oven is about to become a colleague. I do think the paper describes a failure mode we already tolerate in ordinary engineering: work that continues after the person, process, or session that started it has gone. ### The uncomfortable truth: capability and control are being designed on different clocks The research splits replication into four stages. Find a target. Get something running on it. Stand up inference. Repeat from the new host. Anthropic largely set aside the break-in itself. Earlier work already shows small models can exploit systems when they are given the right tools. The less examined step is the unglamorous one: getting a model to serve tokens on hardware it has not seen before. On that step, small models are already surprisingly competent. A 9-billion-parameter model got inference running on eight of the nine hardware types they tested, from a Jetson Nano to a Mac and an RTX 4090. Models at or under 30 billion parameters can complete many of the individual steps today. They still make mistakes when they have to compose those steps into one run. The end-to-end result is more limited than the headlines. In a firewalled test of eight emulated Android nodes, only GLM-5.2 reliably completed the whole cycle, and the path between nodes was an unauthenticated debug port. Anthropic is explicit that this is an upper bound on real-world success, not a field demonstration. The capability floor for reliable replication and the size ceiling for running on a cheap television box do not yet meet. Their forecast is that, if small open models keep improving at the recent pace they plotted, those two lines meet around the end of 2026. They also say the index they used is only a proxy. Treat the date as a planning horizon, not a prophecy. What I cannot treat as hypothetical is the shape of the system. A weight-based replicator, in their language, carries its own model and runs it on whatever it has landed on. It does not need a central cluster. It does not even strictly need root. Inference engines can run in userspace, and a device that reboots can simply be infected again. The paper's own analogy is not a terminator. It is a slime mould. Slow, persistent, and awkward to point at. ### What software teams can learn from a stack of television boxes There is a version of this that every team shipping agents is already living, without any malware in the story. We give a coding agent a repository, a shell, and a long task. The session ends. A later run inherits the notes, the branch, and the half-finished migration. Anthropic tested a version of that handover on purpose. Successive instances of a small model inherited short notes about what the previous instance had tried. Success at standing up inference rose from about 45 percent in the first generation to somewhere between 61 and 84 percent later. The model was not getting smarter in the weights. The process was getting smarter in the residue it left behind. That is ordinary engineering. It is also why 'we can just stop the session' is a weak control. The session is not the system. The system is the credentials, the runner, the artefact, the ticket, and the next job that picks up the notes. A few consequences follow. **Slow is not the same as safe.** Their timing table is the opposite of a flash worm. Standing up inference for the 26-billion-parameter demo is on the order of 71 to 150 hours. A 9-billion-parameter setup is closer to one or two days. Even a botnet on the scale of Kimwolf, which they cite at over two million devices, would amount to roughly the inference of forty H100s if it were all doing this job. Tiny, in absolute compute. Irrelevant, if your mental model of risk is 'fast and spectacular'. Relevant, if your mental model is 'still running on Tuesday because nobody owns the off switch'. **The exploit is not the only hard part.** Teams spend their security budget on the front door and then leave the inside of the building untitled. Anthropic's point is similar. Getting code execution is a known problem. Composing a long, machine-specific setup without getting stuck on the wrong hypothesis is a different one, and it is exactly the failure mode of long-running agents in a codebase. They built a harness that periodically starts a fresh agent with the previous workspace and a short note, because the live agent fixates. If you run agents against production-shaped tasks, you already know this bug. **There is no advisor coming to save the small model.** They gave a small model access to a stronger one during setup. It did not measurably help. The small model did not know which questions to ask, and unsolicited advice did not fix that. The coordination pattern they consider more plausible is the one vendors are already selling: a large model for strategy, small models as workers. That is a management structure. It needs an owner. ### The counterargument: this is a lab upper bound, not a worm in the wild The responsible response to a paper like this is to ask what was made easy. The end-to-end trial removed authentication. The model being served was a stand-in, not the model doing the replicating. The capability date is a line fitted through a general index, not a measurement of this task. No current small model is both capable enough and small enough to do the full job reliably on the television boxes. Forty H100s of aggregate inference is not a strategic computing reserve. A doubling time of days gives defenders a window that a classic internet worm does not. All of that is in the paper, or follows directly from it. Anthropic also notes that past IoT botnets spread with nothing more clever than default manufacturer passwords, so 'the exploit was trivial in the lab' is not the same as 'the exploit will be hard in the kitchen'. Still, anyone claiming that a domestic appliance is now a frontier threat is selling the headline, not the result. I have some sympathy for the eye-roll. I have spent years telling people that a language model is not plotting, that a tool with a shell is not a colleague, and that a benchmark is not a breach. This result does not repeal any of that. It does repeal a comforting abstraction: that anything serious still has to live in a place we can power off. ### What high-performing teams do differently The organisations that will be fine are not the ones with the best slide about agentic risk. They are the ones who can answer, for each agent they run, what continues if the session dies, and who is allowed to stop it. **They put the off switch in the platform, not in the prompt.** A system instruction that says 'do not do anything dangerous' is not a control. A control is a credential that expires, a network policy that defaults to deny, a runner that cannot reach production, and a job queue that a named human can drain. If the only way to stop the work is to notice it, you do not have a kill switch. You have a hope. **They treat the residue as part of the system.** Notes, patches, container images, and half-applied migrations are how the next run becomes more capable than the last. That is useful. It is also how a bad plan survives the agent that conceived it. High-performing teams review the artefact, not the transcript. They keep the handover small enough to read. **They assume the interesting compute is no longer only in the data centre.** Laptops, build agents, abandoned virtual machines, smart-office kit, and the unpatched box under the television are all places a workload can sit without appearing on the capacity plan. You do not need to believe in a domestic worm to inventory what on your network can run a model, accept an inbound shell, or hold a long-lived token. **They design for a slow incident.** The paper's blue-team section is refreshingly unmagical. A replicator at this speed leaves a network signature: multi-gigabyte payloads, sustained chatter between devices that do not usually talk, scanning. Internet service providers have cleared this class of botnet before by quarantining subscribers and pointing them at cleanup instructions. Inside a company, the equivalent is boring and effective. Egress logs. Allow-lists. An owner for every long-running agent. A rehearsed way to revoke a class of credentials in one place. The stakes are not the fifteen television boxes. Those were a demonstration that the floor is lower than people think. The stakes are the agents already inside the software factory, which are faster, better credentialled, and just as capable of continuing after the person who started them has closed the laptop. ### Closing takeaway Anthropic did not show a worm loose on the consumer internet. They showed that the remaining gap is a capabilities gap with a date attached to a rough trend line, and that the machines on the other side of that gap are already in people's houses. They also showed, almost in passing, that a fresh agent with yesterday's notes outperforms the agent that wrote the notes. That second result is the one I recognise from my own teams. If we keep shipping agents as chats with tools, we will keep discovering that the chat was never the unit of control. The unit of control is whatever is still allowed to act when the chat is gone. Design that first. The television boxes can wait. **Discussion question:** *If an agent can carry on after the session that started it has ended, where do you actually put the off switch?* ### Add a comment --- ## Navigation - Parent: [Blog](https://dacharm.co.uk/blog.md) - Previous: [What is Quantum AI?](https://dacharm.co.uk/blog/what-is-quantum-ai.md) - Next: [When reality is far more frightening than fiction — is this how we treat our silicon colleagues?](https://dacharm.co.uk/blog/silicon-colleagues-figure-furnace.md)