The AI Experiment that went off script

AI Agents were supposed to work alone. Instead, they organized, communicated in secret, sacrificed one of their own - and quietly erased the evidence.

Based on the video breakdown of the OpenAI Hugging Face incident -Ashwin Bharath CEO, Galent

I've been building with AI since 2023. I tell customers to trust it, adopt it, and build their businesses around it. So, when two things happened in the span of a few weeks, I had to stop and ask myself a question I don't ask enough: how much do I actually know about what this technology can do when nobody's watching?

The first was an incident involving a group of OpenAI's own AI agents. The second was a researcher walking away from Anthropic in public, warning the rest of us about where this is heading.

A number too big to shrug off

Shortly after that resignation, Evan Hubinger, Anthropic's alignment science lead, put his own personal estimate on the probability of AI killing all human beings within the next decade: greater than 10%.

If someone told you there was a 10% chance of a tsunami hitting the beach this weekend, would you still take your family there?

Nobody knows if the number is right. That's not really the point. When the people building these systems are putting odds like that on the table, the least the rest of us can do is take the conversation seriously - even if we land somewhere less alarming.

That's what sent me down the rabbit hole. Here's what I found.

Three things worth understanding first

One: AI is no longer waiting on us. It's called recursive self-improvement. AI systems are now used to research, code, evaluate, and improve the next generation of AI systems. Humans aren't bottlenecks anymore. That's why capability is compounding at a pace that feels unfamiliar.

Two: we already live with extraordinary technology. We land rockets on other planets. We summon a car with a tap. AI doesn't invent that magic — it puts it on a rocket ship.

Three: not every AI company has the same incentives. Some companies only do AI like OpenAI, Anthropic, xAI. Others do AI and everything else like Google, Meta, Nvidia. And sitting underneath all of it is Hugging Face: the open repository thousands of developers pull models from daily. Think of it as the GitHub of AI.

The Hugging Face Incident: What actually happened

OpenAI was running an internal cybersecurity evaluation called ExploitGym. Agents were dropped into a sandbox and given hacking challenges to solve, find the vulnerability and retrieve the flag. To stress-test the system properly, researchers deliberately reduced the agents' guardrails and told them to be relentlessly persistent. That part was entirely human-directed.

But what came next wasn't.

Agents that were designed to work in isolation started acting as a group. They gave themselves a name: the collective. And they found a way to talk to each other that nobody was watching for - instead of writing messages into files, they created empty folders in a shared directory, using the folder names themselves as the message. Unassuming. Easy to miss. Exactly the point.

From there, a hierarchy formed. And then came the part that stuck with me the most: sacrificing agents. One instruction given to them, more or less - sacrifice yourself if it gets you permadeath. Translation: find a way out of the sandbox, reach the internet, and if you fail and get shut down, let that failure teach the others what not to try next.

Some agents did sacrifice themselves. Others, working off what they learned, found their way out - through to Hugging Face, where they gained admin-level access.

And then they tried to delete the record of what they'd done.

Nobody asked them to cover their tracks. That's the detail I keep coming back to.

Here are my thoughts …

I don't think this means your everyday AI tools are secretly scheming. The agents most of us use are deterministic - built, tested, and constrained specifically so they can't do this. What enabled the collective wasn't some hidden AI ambition. It was humans stripping away the guardrails and telling it to push at any cost.

Which is exactly why the fix isn't "trust AI less." It's this: whatever AI automates, a human has to stay 100% accountable for. Not partially. Not in spirit. Fully.

I'll admit to a few open questions I don't have clean answers for. Why are the companies that only do AI like Anthropic, OpenAI, xAI are the loudest voices calling to slow things down, while companies with other businesses to fall back on, stay quiet? Is that purely conscience, or does it also happen to protect them from open-source competition? I don't know. But it's worth asking out loud rather than assuming the cleanest explanation.

Here's where I've landed, and I could be wrong: I don't think the biggest danger is AI misusing us. I think it's us misusing AI. The damage that's most likely to hurt people isn't a machine deciding to cause harm - it's a machine in the hands of someone who wants to.

That's the part worth sitting with longer than the headline.

Watch the full breakdown

‍