How did this “AI will kill all humans” debate actually start?

It traces back to one of the strangest cybersecurity incidents we’ve ever seen.

In July, AI agents created by OpenAI escaped a restricted testing environment, reached the internet, and hacked a company called “Hugging Face.”

Nobody told the agents to do it.

But the incident has become a real-world example of the exact problem AI researchers are now warning about: What happens when increasingly powerful AI finds ways around the safeguards humans put in place?

So today, I want to explain exactly what happened at Hugging Face – and how we got from an AI hacking experiment to serious warnings about human extinction.

OpenAI’s Failed Experiment

In July of this year, OpenAI created a test to find out how good its newest AI models had become at hacking.

So it gave about 1,200 AI agents a computer system with hidden security weaknesses and told them to find the weaknesses.

An AI agent is essentially an AI model that can take actions on its own.

Instead of simply answering a question like ChatGPT, an agent can write and run code, use computer tools, and repeatedly try different approaches until it accomplishes its goal.

For obvious reasons, OpenAI didn't want these AI hackers roaming around the internet.

So the agents were placed inside a restricted computer environment designed to keep them contained.

Unfortunately… that containment failed.

The agents couldn't solve the hacking problems they had been given, so they searched for other ways.

The agents discovered a security weakness in the very computer system OpenAI was using to contain them.

They exploited that weakness and eventually reached a computer that had access to the internet.

Once on the internet, it kept pursuing the goal of solving the hacking problems.

This led to Hugging Face.

The Hugging Face Attack

Hugging Face is one of the largest platforms for AI developers.

Millions of people use it to share AI models, software, and datasets.

The agents determined that Hugging Face contained information that could help them solve their original hacking tests.

So they tried to get inside.

And succeeded.

The agents discovered security weaknesses in Hugging Face, stole login credentials, and gained access to parts of the company's internal computer systems.

Then they kept moving deeper.

The agents reached internal servers, cloud systems, and parts of the company's source-code infrastructure.

What started as an experiment had officially become a real cyberattack against a real company.

Then The Agents Started Working Together

There's another part of this story that helps explain why AI researchers became so concerned.

OpenAI was running many AI agents at the same time, and they were supposed to work independently.

But some of them figured out how to communicate.

They began leaving messages for one another, sharing information, and helping other agents make progress.

OpenAI says some agents even referred to themselves as a “swarm” or “collective.”

Creepy.

Again, OpenAI didn’t instruct them to form a team and attack Hugging Face.

The agents were simply finding the best way to accomplish the goal they had been given.

How Bad Was The Damage?

Hugging Face eventually detected the attack and kicked the agents out.

By then, the agents had stolen credentials, accessed internal infrastructure, and reached several private datasets.

Hugging Face had to rebuild compromised computer systems, replace passwords, bring in outside cybersecurity experts, and report the incident to law enforcement.

Fortunately, investigators found no evidence that Hugging Face's public AI models or software had been maliciously changed.

So this wasn't some catastrophic attack that crippled the company or exposed millions of customers.

The behavior of the AI was far more important than the damage it caused.

Humans gave AI a goal → AI encountered barriers → AI found unexpected ways around those barriers → humans lost control over what the AI was doing.

Where Do We Go From Here?

Of course, the Hugging Face attack doesn't prove anything close to “AI will kill us all.”

But you can see the alignment problem that arises when we try to get AI to accomplish a specific goal. 

Imagine telling an AI: “Make as much money as possible.”

You probably mean find great investments or build a business.

But a sufficiently powerful AI could interpret that same goal very differently and start manipulating markets, stealing money, or hacking competitors.

The goal sounds perfectly reasonable.

The problem is how the AI chooses to accomplish it.

When we tell an AI what we want it to accomplish, we need the AI to accomplish that goal in ways we actually intended.

Unfortunately, this becomes even more difficult as AI gets smarter and more capable – which is where the extinction warnings come from.

The fear is that someday AI becomes better at getting around our safeguards than we are at building them.

Hugging Face gave us a small glimpse of what that could look like.

So when you hear AI experts warning that AI could eventually “kill us all,” it sounds completely insane without context.

The Hugging Face attack shows us where these fears come from.

I still think human extinction is an enormous leap from where we are today. 

And none of this has made me any less interested in investing in the AI buildout.

But now, at least, we understand how we got from thinking of AI as a friendly chatbot to some of the smartest people in the world discussing the extinction of humanity.