Back in the early days of the Internet – 1988 to be precise – the self-replicating computer program known as the Morris worm spread across the network and shut down thousands of online VAX and Sun computers. It was a good example of software not going rogue, but doing exactly what its creator had intended it to do without understanding the consequences. As Wikipedia explains:
By instructing the worm to replicate itself regardless of a computer’s reported infection status, Morris transformed the worm from a potentially harmless intellectual and computing exercise into a viral denial-of-service attack.
Since that early botched experiment, intentionally malicious worms have been written to create botnets that can launch DDoS attacks, for example to overwhelm systems punitively, or for the purpose of blackmail. Examples include Mirai and the later Aisuru-Kimwolf. These worms targeted specific kinds of Internet-connected hardware. But the arrival of AI agents that can carry out actions autonomously brings with it a new threat: AI worms potentially able to seek out, hack into, and infect any device that is online. Researchers at Anthropic have been studying this possibility:
small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes….Unlike non-AI malware, [such a replicator] could also figure out unfamiliar devices on its own.
Previous work has shown that AI models can carry out this kind of attack on more powerful systems:
artificial intelligence (AI) agents enable a fundamentally new threat: a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks. Deployed on a network of machines spanning Linux, Windows, and IoT (Internet of Things) devices, the worm propagated by exploiting common, real-world corporate network vulnerabilities.
The Anthropic researchers explored whether an AI worm could run a capable model in a distributed fashion across low-power IoT devices, as “shards”:
A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model. Machines that could run a significant model shard for a <30B model [that is, an LLM that contains fewer than 30 billion parameters] include smart fridges, smart ovens, automatic vacuum cleaners, and lawnmowers. A growing number of home and business devices, such as security cameras and video-conferencing units now ship with small built-in AI chips, designed to run AI models on the device itself.
A 26B model running across 15 off-brand Android TV boxes, costing around $55 each, created manually by the researchers as a demo system, showed that IoT devices could indeed run quite capable models as shards, albeit slowly – at 0.39 tokens per second. Building on that result, the research project sought to answer two questions:
First, can small models get a model running on unfamiliar hardware? And second, can they do the whole cycle (find an uninfected device, copy themselves over, start running there, and repeat) all together in a single run?
The answer to both is “yes” – details in the Anthropic paper. The good news is that the process is very slow, typically taking one or two days to replicate. As the researchers put it: “As an analogy, imagine the replicator as a slime mold or a swarm of fruit flies, which replicate over days.” However, they also caution that things will speed up as IoT hardware becomes more powerful, and LLMs more compact. They write in conclusion:
As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.
These replicators will not be an example of AI agents “going rogue”, but of humans intentionally exploiting new capabilities for malicious purposes – or perhaps simply releasing badly written software, just like the Morris worm all those years ago.
Follow me @glynmoody on Mastodon and on Bluesky.