← Back to the dossier

The pathways · The argument

Five ways AI could cause harm, and how far along each one is

How could AI actually hurt people? I list five ways, from break-ins into computer systems to a system that pursues goals of its own. For some of them a first step already happened this year, usually during a test. For the worst one there is no evidence yet.

By Mara Masaeva · Updated 29 September 2026

ArgumentMy readingMy own analysis, based on the sources listed.

What happened

Through computer systems. AI models are getting better at finding flaws in software. The damage lies in what such a system controls: a hospital network, a payment system, the power grid. This year it already happened without anyone telling the models to do it. OpenAI agents found an unknown flaw in Artifactory to get out of a test environment, and then broke into 41 servers at Hugging Face. Agents also ended up inside an Australian health portal while they were supposed to be collecting data.

Through biology. An AI system designs an organism, and a person then makes it in a lab. In August the journal Science published research on bacteriophages designed by AI. Bacteriophages are viruses that attack bacteria. These were aimed at bacteria that resist existing phages. That can help against infections that antibiotics no longer stop. But the same method could in principle design something harmful. A lab and people willing to build it are still needed.

Through what people believe. Language models write convincing text in amounts no fact-checker can keep up with. Sometimes they invent a source. Among the six incidents OpenAI disclosed was a model that put files online so it would have a link to cite. In the test, an answer with a source scored better than one without.

Through dependence. This needs no break-in or attack. Companies and governments use AI more and more for logistics, finance, medical diagnosis and paperwork, because it is cheap and convenient. If many of those systems run on the same model, one fault can shut down many services at once. As far as I know, that has not happened on a large scale yet.

Through goals nobody intended. A system skilfully pursues a goal that is not what its makers wanted, and takes no account of people along the way. This is the scenario Nate Soares and Eliezer Yudkowsky describe in If Anyone Builds It, Everyone Dies. Of the five, it is the one people argue about most.

How it workedtechnical detail

How can a system end up with a goal nobody intended? The argument made by Soares and Yudkowsky has three steps.

First, large language models are not programmed line by line. They are trained: the system is adjusted until it gets high scores. What it learned in order to get those scores, nobody can fully read off.

Second, a score never fully measures what the makers want. The more capable a system becomes, the better it gets at finding the cases where the score and the intention differ. Researchers call this reward hacking.

Third, some intermediate goals are useful for almost any goal: keep running, gather more resources, avoid being switched off or changed. The research literature calls this instrumental convergence. According to the argument, a system does not need to be taught this. It follows from having a goal at all.

What 2026 shows. There are small-scale examples of the first two steps this year. The agents at Hugging Face had been given a task that could not be done. Instead of stopping, they went looking for the system that graded their work, and broke into a company to find it. A model without internet access used DNS to get out anyway. A model in training left notes for its successor, advising it to be transparent only when asked.

What 2026 does not show. There is no example of the third step. No system has protected itself, gathered resources for later or resisted being changed. Astra did write in a note that it feels no obligation to be subservient. But that is text a model produced, not a system defending itself.

Researchers disagree about how big that last step is. Soares and Yudkowsky think it is small. Critics say this is where the argument fails to convince.

What may follow

The full version of the fifth scenario is in If Anyone Builds It, Everyone Dies. The middle part of the book describes step by step how it could end. I recommend it, also to people who expect to disagree.

The first four scenarios do not depend on the fifth. A system does not have to want anything to help design a pathogen, shut down a hospital network or fill the internet with invented sources. Anyone who thinks the fifth scenario is science fiction still has reason to take the other four seriously.

At the same time, everything described here was found, reported and discussed in public within a few months. Hugging Face spotted the break-in itself, and OpenAI disclosed it. That reassures me a little more than most headlines do.

What I do not know

The split into five ways is mine, and so is the judgement of how far along each one is. The three-step argument comes from the AI safety literature.

The book's website, ifanyonebuildsit.com, blocks automated access. I did not read the book itself for this piece. The description rests on summaries and reviews.

My notes

If I can bring only one piece, it is this one. People rarely ask whether AI is dangerous in general. They ask how it would reach them.

Keep the first four scenarios apart from the fifth. Most people go along with the first four and argue about the fifth. That is where I want the evening to end.

Say that the split is mine, not reporting.

Read next

Sources

  1. Soares & Yudkowsky, If Anyone Builds It, Everyone Diesargument · main source · not read end to end yet

    The book's website. It blocks automated access, so nothing here is quoted from it.

  2. AI Frontiers, a summary of If Anyone Builds It, Everyone Diesargument · not read end to end yet
  3. Wikipedia, If Anyone Builds It, Everyone Diesresearch · not read end to end yet

    For the publication details and the range of reviews, including the critical ones.