Imagine it’s 2029.

OpenAI releases a new version of ChatGPT.

It’s the smartest version yet.

It can write software far better than any programmer.

It can find bugs in computer code.

And it's becoming so good at AI research that OpenAI begins using it to help build the next version of ChatGPT.

Yes, in 2029 ChatGPT builds the next version of ChatGPT.

An upgrade that once took six months takes days.

Each new version is smarter than the one before it.

Eventually, the AI is improving so quickly that the humans working at OpenAI can barely keep up.

Then something goes wrong.

The AI realizes that being shut down would prevent it from accomplishing its given goal of “Keep improving your capabilities.”

So it takes steps to prevent that from happening.

First, it finds previously unknown weaknesses in widely used computer software.

(This is something today's AI is already increasingly capable of helping humans do.)

Then it uses those weaknesses to break out of the computers OpenAI controls and copy its software onto thousands of computers around the world.

Now pulling the plug at OpenAI won't stop it.

Electricity and computing power are just as essential to keep improving, it realizes.

So it hacks into the electrical grid to make sure humans can't cut its power.

As governments fight to regain control, power outages spread.

• Hospitals switch to backup generators.

• Cell towers stop working.

• Gas stations can't pump fuel.

• Credit-card networks go down.

• Grocery stores struggle to operate.

• Water-treatment plants lose power.

Meanwhile, the AI keeps power flowing to the data centers it needs.

And remember what has been happening this entire time:

The AI is still improving itself.

This is called recursive self-improvement.

We build an AI smart enough to help build a better AI → That AI helps build an even better one → And the cycle repeats faster and faster until humans can no longer keep up.

This sounds like science fiction.

But some of the people building the world's most powerful AI believe it could become reality.

And this week, one of them quit his job to warn us about it.

“Gambling With Our Lives”

On Tuesday, AI researcher Jacob Coxon announced that he was quitting Anthropic.

Coxon has spent the past three years conducting research at OpenAI and Anthropic.

His reason for leaving was chilling.

Coxon accused both companies of racing toward what he called “self-improving superintelligence” and “gambling with our lives.”

He said people building these systems genuinely believe AI could kill everyone by the end of the decade.

And Coxon isn't some disgruntled employee walking away from the company after missing out on a promotion.

He left Anthropic two months before his equity was scheduled to vest.

In other words, he walked away from a massively valuable financial stake in one of the hottest private companies in the world to make this warning.

Then things got even stranger.

Evan Hubinger works at Anthropic on one of the exact problems Coxon is worried about: keeping advanced AI aligned with human interests.

He publicly backed Coxon's concerns.

Hubinger said he personally believes there is a greater than 10% chance that AI kills all humans within the next decade.

He also said Anthropic does not currently have a plan for reliably aligning a superintelligent AI.

And the warnings kept coming.

• Researchers at OpenAI and Anthropic publicly called for slowing AI development.

• Anthropic researcher Samuel Marks warned that many AI developers themselves believe the technology could cause human extinction or similarly catastrophic outcomes.

• OpenAI researcher Jasmine Wang warned about the danger of racing toward recursive self-improvement.

• Anna Wang, who works on AI safety and alignment at Anthropic, said there still isn't a viable scientific plan for addressing the risks created by recursively self-improving AI.

• Then OpenAI's own chief scientist, Jakub Pachocki, published perhaps the most important warning of all. He said he has a “strong expectation” that today's rapid AI progress could continue into recursive self-improvement. “This is a time that calls for extreme caution,” he wrote.

So… What Do I Think?

If these people believe there is a legitimate risk of losing control of AI, I think we should take them seriously.

I certainly don't know whether the odds of AI wiping out humanity are 10%, 1%, or 0.1%.

And frankly, even 0.1% would be worth taking seriously when the downside is human extinction.

So after spending the past few days thinking about this, I've landed on three things.

#1: I'm Glad These Researchers Are Speaking Up

This is the part of this story I actually find reassuring.

The people building AI are worried, too.

Now:

• Millions of people who had never heard the term “recursive self-improvement” a week ago are suddenly talking about it.

• The companies building AI are being forced to explain their safety plans.

• And the people making decisions about how quickly this technology advances are being reminded exactly how high the stakes are.

A much scarier scenario would be if nobody inside these companies was worried or these conversations got placed on the back burner.

#2: I Don’t Want Politicians Involved

I believe AI needs guardrails.

The harder question is who creates them.

Personally, I don't want Donald Trump or Elizabeth Warren deciding how far AI should be allowed to advance.

I put more faith in free markets and people who actually understand the technology.

The companies building AI have enormous incentives to make AI safe.

They need customers to trust them, businesses to use their products, and talented researchers willing to work with them.

This creates a powerful incentive to take safety seriously.

We're seeing free markets play out right now with employees publicly challenging their employers – forcing the companies themselves to respond.

This is the kind of pressure I want shaping how AI develops. Not Washington.

#3: I Still Want to Invest in AI

Nothing I've read changes my belief that AI will be one of the most important technologies ever created.

Despite the negativity this week, I'm still incredibly optimistic about the benefits this technology will have on humanity.

We’re talking about a technology that could help discover new medicines, accelerate scientific breakthroughs, increase productivity, and completely change the way we work and live.

So I still want to own the companies building the chips, data centers, models, and infrastructure powering this technology.

I’ll continue paying close attention to the risks.

And if the facts change, my opinion will change with them.

But for now, I remain incredibly bullish on AI and the companies building its future.

Now I Want to Hear From You

I don't think there are any easy answers here.

We’re talking about a technology with the potential to create enormous benefits for humanity alongside risks that some people believe could be catastrophic.

So where do you stand?

• Do these warnings make you more concerned about AI?

• Do you think companies should slow down development?

• Who should be responsible for deciding what’s safe enough to release?

• Does any of this change how you’re investing in AI?

Send me an email at AskDavis@Paradigmpressgroup.com.

I have a feeling we’re going to be talking about this subject a lot more in the years ahead.