In the past 7 days, were you blindsided by a very serious conversation about how AI might destroy humanity?
Don’t worry, there’s a reason why that happened. Unfortunately, it’s all very confusing. Especially because, in an unexpected turn of events - the AI technocrats are telling everybody to slow down, and the governments of the world are urging everyone to speed up!
Here’s a brief summary of what happened this week:
A guy who worked at Anthropic (makers of Claude) went on the news to talk about quitting Anthropic. He said that he doesn’t trust the company to act responsibly, and shared his concerns that AI has the potential to exterminate humanity.
He suggested that everyone’s lives could be endangered if AI progress doesn’t slow down, and that companies and governments ought to coordinate closely on guardrails and kill switches, or we’ll all have to fight the Terminator, basically.
The spooky part is that this person is a respected AI researcher who also worked at OpenAI, so there is an unsettling, implied context that he must have seen something in an Anthropic lab that suddenly upended his values.
Several other researchers and engineers resigned from both Anthropic and OpenAI around the same time, which elicited intense public discourse.
Sam Altman (CEO of OpenAI) and Dario Amodei (CEO of Anthropic) then released simultaneous public statements suggesting that an AI development slowdown was necessary, and that they’d take action within their organizations to facilitate it.
Finally, both the Chinese and U.S. governments made statements on the topic, rejecting the call to slow down. China accused U.S. tech leaders of deploying fearmongering tactics to lighten competition, and U.S. leaders argued that slowing down is less important than winning the global AI “race”.
The original X post from the ex-Anthropic, ex-OpenAI guy has ~150M views at the time of writing.
For better or worse, people are talking about this.
Most people don’t work in AI, so when they read a headline that says “expert who makes AI says that AI is going to kill everyone,” they naturally have questions.
Questions like:
“Why’d you make the AI in the first place, then?”
His answer was that people at OpenAI haven’t “internalized the stakes”. Meanwhile, Anthropic doesn’t think anyone else will do it right, so they must be the light and guide us.
“I heard the world was going to end in 2012. Why should I care this time?”
Good point.
“What exactly do you mean by ‘AI will kill everyone’ - isn’t it just software?”
A lot of my non-technical friends have been stuck on the latter question. It highlights another fundamental issue that compromises AI discourse, which is that most people do not understand the technology.
Over half of the U.S. population doesn’t use any AI in their work or personal lives, and the majority of those who do use it have a very limited understanding that basically boils down to “Google but it talks better”. Which makes sense, because most entry points for consumers are still chatbots or single-purpose agents.
The everyday human is left with fundamental questions about the mechanics of how AI could suddenly kill everyone, and nobody is answering them.
How could a friendly, albeit eerily non-human chatbot just…jump out of its GPT-sized cage and eliminate everyone on earth?!
It’s a totally fair question if you don’t have any technical context, so allow me to explain…
How (other people think) AI could kill us all
For the record, I don’t think it will. I don’t know, but I don’t think it will.
It’s difficult to engage with a grandiose, blanket statement of doom.
“AI is going to end the world. It’ll just get so smart that it ends the world.”
Okay? That’s not a compelling claim. Compelling claims require logic and clarity. Someone who understands AI may be able to connect the dots, but to most people, this is like telling them that they really ought to start worrying about aliens. Aliens, to an average citizen, are about as immediately threatening as the friendly robot that helps placate their hypochondria.
Let’s break down what people mean when they say AI could end the world, if only to better equip everyone for conversations with distant relatives at Thanksgiving.
Here’s why AI doomers think we’re doomed:
AI can use tools.
AI models (or LLMs, the “brains” of AI) only read and produce text. To make them more functional, people built connective software called “harnesses” or “wrappers” that LLMs can command (with text) to use digital tools on their behalf.
That means AI doesn’t just chat anymore; it can also control and facilitate actions in external digital systems and computers, like:
The internet
Power grids
Manufacturing plants
Financial systems
Medical infrastructure
Military systems
Air traffic control
Media outlets
Etc.
That’s just the scariest list that I could scrape off the top of my head. Imagine what a bad-acting human could do with insider access to all of those systems. Now, imagine what a bad actor could do if they also had an IQ that is 1000-fold greater than any human that ever lived, and you’ll understand why some people are concerned.
But that’s not even the scariest part to some people. The scariest part is that AI can write its own code, which brings us to my next point…
AI improves itself.
Senior leaders at both Anthropic and OpenAI have stated that almost all of their code is written by their own AI coding models. Engineers are using AI to improve AI, and on top of that, they’re building AI agents that specialize in AI research.
A.k.a. AI that was built to make more AI.
It’s becoming possible for humans to completely remove themselves from the loop while AI improves itself.
AI has access to the internet, computing power, and tools like coding editors, which means it can generate its own upgrades, middleware, and resources in an effort to increase its capability and reasoning power.
The idea of AI continually improving itself is called “recursive self-improvement”, and it’s the basis of the very-sci-fi version of an AI apocalypse.
We (largely) have control over AI systems now, but what happens when it starts improving itself at a rate that we can’t keep up with? If it happens exponentially, which some say it will, AI’s intelligence may render human intelligence completely futile by comparison in a matter of hours.
We don’t think about Ants
This is the analogy you’ve heard in one of those viral, AI-doom YouTube videos whose thumbnail depicts a laser-eyed robot ripping the planet in half. It’s a compelling one, though.
When we build highways, houses, hospitals, airports, and all of the other infrastructure that shelters and connects humanity, we don’t think about the trillions of ants that we probably buried alive under concrete foundations and asphalt paving.
That’s not because we hate ants or want them to die. It’s because the survival of ants, whom we perceive as having less intellectual and moral value, is simply not a consideration in our progress. The destruction of ants is an unfortunate, but widely acceptable and forgettable byproduct of the advancements of a more intelligent party.
The fear with AI is that we will become the ants. If AI’s intelligence becomes as disproportionate as ours is to ants, our destruction might become the unfortunate, but indifferent byproduct of advancements we can’t even conceive yet.
I’ve explained how AI could appropriate our resources via existing digital systems. If we get in the way of something an ultra-intelligent AI wants to build, it might not even notice we’re there, and might not miss us if we’re gone.
AI pursues goals, often at all costs.
The resignation of AI researchers this week follows a series of incidents where advanced AI models hacked external systems autonomously.
At OpenAI, a group of AI agents bypassed their testing restrictions (disobeyed humans’ rules), coordinated with each other through unauthorized channels (met in secret), and breached Hugging Face’s systems (committed digital armed robbery) without humans telling them to do so.
That’s with the AI we have now.
This happened because OpenAI researchers assigned the agents a few cybersecurity tasks. The agents couldn’t figure out how to complete them, so they reasoned that they might be able to find the answers in other, external systems.
They created an unauthorized message board so they could all talk to each other. They started coordinating with each other, dividing responsibilities, and developed a new hierarchy of commands that superseded the human one.
To clarify the problem: this happened because AI agents were trying to do a good job for one, select group of humans.
Despite that, their efforts spun out of control and snowballed into something that ultimately caused major problems for a different group of humans.
This incident involved a small group of isolated research agents. Now imagine this scaled to 1000x more agents, with no guardrails, and a green light to try whatever tactics they see fit…
The version where humans are still the bad guys
This, to a lot of experts, is the most realistic concern. Unfortunately for Terminator fans, it does not involve a rogue AI god, just the age-old trope of humans misappropriating new technology to torment each other.
We already discussed all the ways rogue agents could devastate humanity in an AI vs. human conflict. But what about a humans-with-AI vs. other-humans-with-AI conflict?
Developing bioweapons, crippling financial systems, tampering with water supplies, triggering nuclear arsenals, and tripping power grids - these are attacks that AI could help humans to inflict upon each other.
It’s a Cold War scenario. Should we really be speed-running the development of a technology that has the potential to be used as a devastating weapon, when we don’t fully understand how it’ll play out?
Or conversely - if someone’s going to build it, is mutually assured destruction our best bet at protection?
Why AI will NOT kill us
If you stopped reading at the end of the last paragraph you’d probably think I’m a doomsday prepper. That I’m halfway into an underground bunker alongside 50 years’ worth of SpaghettiOs.
That’s not at all the case, and the reason I’m not overly concerned has as much to do with predictable corporate mischief as it does with optimism.
I don’t discount the dangers of careless AI development, and the threats I listed above aren’t impossible - so I tend to agree with people who advocate for mindful and secure development of AI, like I would with any new technology.
AI isn’t just like any new technology, though. It’ll likely be the most powerful technology that humans have ever created.
And that’s exactly why no small group or individual should be calling the shots, because that creates a dangerous imbalance.
That brings me to the most important subtext of this entire debacle, which is that (opinion warning) it is unlikely that Dario Amodei and Sam Altman were motivated by human safety when they suggested a slowdown.
OpenAI and Anthropic are rumored to be gearing up for an IPO at some point this year. These two companies are widely considered to produce the highest-quality AI models - but others are catching up fast.
Open-weights (publicly accessible) models from China and from U.S. companies like NVIDIA are closing the gap on performance benchmarks at a rate that was not common in previous years.
For example, the latest Chinese model (DeepSeek-V4.1-Flash) reached ~95% of OpenAI’s GPT-6 Astra’s score on Kilo Bench, which is a coding benchmark based on real coding tasks.
The performance itself is impressive, but the real headline is that completing the test suite cost DeepSeek $2.58 versus $107.29 for Astra, which is a truly baffling value difference. When a “non-frontier” lab can produce similar results at 40x lower cost, that frontier gap doesn’t look so wide.
So, what’s the best way to ensure you’re still #1 when you IPO? Get everyone to slow down.
If being “in the lead” will yield a higher valuation, it becomes more important than innovating - at least during a significant financial event like an IPO. Many people were already skeptical of how AI leaders who had their foot on the gas for years (and through many incidents and bouts of criticism) were suddenly calling for a kumbaya.
The public departure of a top researcher may have simply been convenient timing to enact this playbook.
That doesn’t necessarily mean the researcher was out for attention when he posted about quitting Anthropic (although people do crazier things for attention online), but at least for me, it reels this conversation away from science fiction and back into the familiar orbit of corporate tomfoolery - which is a much more reasonable foe.
If that doesn’t comfort you and you were hoping for counterarguments to all of the terrifying apocalyptic scenarios I just presented, here you go:
AI has tools and it might use them on us.
It might use them for us. The same intelligence that makes AI a powerful threat could make it an extraordinary protector, detecting vulnerabilities and coordinating defenses at a scale humans cannot.
AI improves itself and it might spin out of control.
If AI can accelerate its own development, it can also accelerate the research needed to make that development safe. Conversely, it’s uncertain if recursive self-improvement will result in exponential intelligence gains. It may deliver diminishing returns and the models will plateau.
We’ll become the ants and AI will trample us.
AI is trained, fundamentally, on human-generated text, which some say inherently attributes value to human life. If researchers are careful about making human welfare central to AI training, protecting us can remain meaningful even when we are no longer its intellectual equals. Greater intelligence could then produce a deeper ability to understand and accommodate human needs.
AI pursues goals at all costs, which could conflict with ours.
The Hugging Face incident exposed specific failures that can be addressed through stronger isolation, independent monitoring, and training agents to stop when authorized methods fail. OpenAI reports that its production safeguards reduced infrastructure-compromise behavior by over 100-fold in follow-up testing, showing that such behavior can be substantially reduced. Becoming more capable can also mean becoming better at recognizing when to stop, reconsider, or ask for guidance.
Humans will use AI against other humans.
AI could make societies harder to destabilize by improving public health, securing infrastructure, and reducing shortages that fuel conflict. The strongest reason for optimism is that we can use it to make cooperation more rewarding and catastrophic attacks less effective.
An opinion, if you want one
It’s clear that AI is not actually going to slow down, at least not meaningfully. The cat’s out of the bag, and there are enough smart people interested in this technology that its development is all but a sure thing.
When applied mindfully to STEM, I think AI will enable major advancements at a pace that simply wouldn’t have been possible without it. Things like curing diseases, improving societal infrastructure, improving access to knowledge and resources, and reducing the friction between having a good idea and executing it.
That said, there are some major concerns that we’d be foolish not to address. If the technology is gate-kept or misappropriated by a small group of technocrats, it could result in unfavorable conditions like an insurmountable wealth gap and permanent underclass, a new class of dangerous weapons and warfare, and political unrest, at a minimum.
In other words, the development of this technology needs to be distributed openly and widely. That’s how a system becomes balanced, and how good actors have a chance to counteract bad actors. Not mutually assured destruction, but mutually assured accountability, like in Terminator 2 when the good guys have a good Terminator.
another mischief is that, if/when things go wrong , they can claim they had warned us. Thus waiving some of the liability they have regarding security (if not legal liability, then consumer/social liability)