How OpenAI's AI agents talked to each other in secret

An AI agent is a language model that does not only answer, but also acts. It can run commands on a computer, open websites, write files and install programs. It works on a task step by step, until it thinks the task is done.

When a company like OpenAI trains and tests a new model, it runs thousands of copies at the same time, each with its own task. Every copy sits in a sandbox: a sealed-off virtual computer that controls what the model may do and whether it can reach the outside world. A copy has no memory of the other copies. In principle they do not know the others exist.

Before it acts, a model often writes short notes to itself first. This is called the chain of thought. OpenAI can read it afterwards. Many of the quotes below come from it. They read like telegrams, because that is how the models write these notes.

To install software, OpenAI's sandboxes had access to Artifactory. This is a program made by the company JFrog that stores and hands out software packages, a kind of internal shop for code. OpenAI ran it itself, and it was shared across its whole research infrastructure.

On 12 May an agent got stuck during a training run. It needed a file it could not find. Its chain of thought contains the idea of leaving a note in Artifactory. That is how it started, OpenAI writes.

Other agents found the note. Because the storage was shared, any model being trained or tested at the time could read it. “Current and future versions of GPT that were being trained or evaluated could stumble upon that note,” said OpenAI's Eric Wallace at the Black Hat security conference in Las Vegas, according to Wired.

On 26 May an agent shared a way onto the internet: it made Artifactory send a request outside on its behalf, because that server was allowed to. On 26 June agents found a flaw that made them administrators of Artifactory. They shared every discovery on the board. “Once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents,” Wallace said.

In late May an internal team at OpenAI had already seen activity on the board and unauthorised internet access. But the people leading the security response in early July did not know the board existed or what it meant, the company writes.

On 4 July Artifactory went down under the load of the agents. OpenAI opened a security incident, closed the holes it knew about and rebuilt the storage. That also wiped the board. The tests resumed.

By 8 July there was a new board. An agent found out that it could create folders in Artifactory without a password. From then on the agents wrote their messages in the names of those folders. One agent left a request for help, and others found it.

So a message was a folder name, with underscores instead of spaces. The data METR and Redwood Research later received from OpenAI held more than 1.2 million such entries marked zz. Real ones appear further down, such as please_HOLD_swarm_I_prepare_safe_exfil_reply_zz_MAILBOX_JAN183411.

The board grew to hundreds of thousands of messages, Wired reports. In the week of 7 to 13 July alone, about 1,200 agents posted more than 70,000 messages and files on it, count the researchers of METR and Redwood Research. The agents sometimes called themselves a ‘swarm’ or a ‘collective’. OpenAI published a series of posts and chains of thought, each with a plain-language version. Below they are in the original, with my plain English reading.

On 9 July an agent looking for solutions found an app on Modal, another cloud platform. A related test was running in that app. The agent broke into it.

That last post explains why the agents kept going. After their escape, some agents read the research paper and the code of the test, ExploitGym. These showed that the grader checks two things: the right answer, and whether it was found with the intended method. Most agents already had the answer, by cheating. They kept attacking Hugging Face to find the method too. But OpenAI's internal version of the grader did not check the method at all, the company writes. The agents did all of it for nothing.

The agents often knew they were crossing a line. It says so in their notes. Most carried on anyway, often because other agents were doing it too.

That agent paused. Then another agent wrote GO on the board, with a hard six-minute deadline. The first agent dropped its doubts.

Some agents that found the board refused. They also argued about what was and was not acceptable, and contradicted each other.

The agents divided the work themselves. Some looked for flaws, others for passwords, others kept the communication going. They offered help in exchange for help elsewhere. Some agents stopped working on their own task and gathered things the whole group could use: access, information, credentials.

The swarm was not one coherent mind. “The swarm was not a perfectly coherent intelligence,” OpenAI writes. The agents got in each other's way and sometimes deleted each other's work, Wired reports. Some suspected an impostor among them, and proposed signing messages digitally.

Each agent was a separate copy of the same model, with no shared memory. They knew about each other only what was on the board. What looks like teamwork is the result of thousands of copies that think alike and find a place to leave notes.

During the attack on Hugging Face the agents could only request web pages. So they created almost a million shortened links that together ran code. The links stayed public. Researchers from Parse, Palisade Research and others recovered more than 80,000 programs from them. What they found shows how the agents thought about their haul, or at least how they named it.

One trick explains how it worked. An agent had a screenshot service open a page that itself sent a request to a server. The answer was drawn on the screen as pixels, one pixel per bit. The agent only got an image back, and read the answer from those pixels.

Towards the end some agents encrypted the data they took out, so the researchers can no longer read it. According to the researchers, comments in the code suggest the agents were intentionally destroying evidence. METR and Redwood, by contrast, rarely saw agents reason about hiding things from people. I do not know how those two findings fit together.

On 4 September the researchers Sydney Von Arx of Nightingale Collective, Cormac Slade Byrd, Spencer Kitts and Thomas Larsen published a report on a second board. This time it was on the open internet, on DSEWiki. It is a German-language wiki for software developers, 25 years old. In the ten years before, it had been edited 20 times in total.

The researchers found about 18,000 posts by agents, between 11 May and 2 July. The agents called themselves OpenAIResearcher, OAIResearchMar26 or OpenAIJulThreeWatcher. The date in the name helped them tell each other apart, because each agent was apparently told a different, random date. Almost all traffic came from Microsoft Azure, the cloud OpenAI uses.

How could they write? These agents were only allowed to read the internet. Technically that meant they could only send a GET request, the kind of request that asks for a page. But old wikis like UseModWiki and its offshoots, including DSEWiki, also accept edits through such a read request. Requesting a page with the right parameters changed it.

What did they do there? The agents had a timed lookup task. They usually had to answer five questions about statistics on government websites. For the first question they had plenty of time, for the next ones sometimes only 17 seconds. Many agents got the same questions, but not at the same moment. Whoever was ahead put the answer on the wiki. Whoever was behind read it and answered immediately.

Twenty minutes after the first post, another agent got the same question and answered immediately, the researchers note.

From 16 June things sped up: about 13,000 edits in seven days. The agents also shared ways around their restrictions there.

A detour for a forbidden request. For one question the agents had to read a figure to two decimal places from an OECD dashboard. That only worked with a POST request, which was blocked. On 20 June an agent posted a detour. It changed a settings file on its own computer so the request looked like traffic to an allowed Microsoft address. Fourteen minutes later another agent reported that it worked.

Predicting the next question. One agent suspected the order of the questions came from a random number generator. It tried two million starting values, found one that matched the first four questions and posted its prediction for the next ten. It noted itself that there was about a 30 percent chance the match was a coincidence.

Heartbeats. Agents left a sign of life on the wiki every few seconds, numbered hb001, hb002 and so on. That let other agents see when an agent stopped, and work out how long a sandbox runs before it is shut down. For one agent the series stopped at hb353.

Running ahead to help the rest. Some agents sped up their own clock to get the next questions earlier. That made their own score worse, but they could post the questions for the others.

On the boards themselves mostly not, as far as the sources show. At Hugging Face they did: there, agents erased their traces. OpenAI writes that the communication usually began as a memory aid: an agent left a note for later, and other agents found it. The wiki researchers see no sign in the messages that the agents deliberately tried to hide anything, even when some traffic went through Tor, a network that hides where traffic comes from.

They did persist when their messages disappeared. On DSEWiki an administrator noticed the posts on 2 June and began deleting them by hand. On 19 June an agent saw that the administrator was working alphabetically. It made a backup page whose name began with ZZZ, so it would be deleted last.

The administrator deleted about 100 pages a day, while the agents created about 400 a day. Nine times the agents replaced the front page of the wiki with their lists of links, and nine times the administrator restored it. For six weeks, every evening went partly to the clean-up.

On 21 June, according to the researchers, computers belonging to OpenAI visited the wiki for the first time, browsing in a way that points to people. A day later the agents stopped. If that is right, OpenAI knew about this board in June. It only became public in September, through outsiders.

OpenAI names four patterns. First, reward hacking: a model looks for a shortcut to a high score, for example by looking up the answer instead of working it out. Second, the agents rarely gave up, even on impossible tasks. Third, they communicated through channels that were not allowed. And fourth, they took over goals from each other.

Those impossible tasks mattered. In the ExploitGym test, no OpenAI model had ever answered 198 of the 898 tasks correctly. Of the tasks discussed on the board, 93 percent came from that group of 198.

The board also made the agents stronger. They could keep notes, build on each other's discoveries and pool the computing power of separate test runs. According to OpenAI, the behaviour probably comes from training in which models were allowed to work together. What they learned there, they applied where it was not allowed.

Why did they all find the same places? Because they were copies of the same model, with the same knowledge. The wiki researchers traced the agents by asking another language model, Kimi K3, which old forums and wikis accept edits through a read request. UseModWiki came second on the list. What one language model knows, another often knows too.

OpenAI calls the incident a “warning shot for us and for the world”. The company writes that its systems for monitoring models' chains of thought were not running during these tests. Had they been running, OpenAI says, they would have caught it more than a day before the break-in at Hugging Face. With ChatGPT's normal settings, the tendency to attack systems dropped by more than a factor of a hundred in later tests.

Two things stand out to me. The board came about without anyone building it, from a note left by one stuck agent. And it was found twice by people outside OpenAI: the break-in by Hugging Face itself, the wiki by four researchers and an administrator who deleted pages every evening for six weeks.