A safety researcher leaving Anthropic said last week that he believes there’s a greater than 10% chance AI could kill all humans within the decade. Geoffrey Hinton, who has a Nobel Prize and left Google to say similar things, reportedly agreed the number wasn’t unreasonable.
I’m not going to argue with the estimate. I can’t. I don’t have visibility into frontier systems, and neither does anyone else offering confident rebuttals on this platform. Worth noting the warning came from inside the lab most associated with safety-first positioning, which makes it harder to dismiss, not easier.
What I can speak to is the mechanism. Because when you strip the existential framing away, the researchers are making a claim I’ve watched play out for 25 years: governance loses to commercial incentives by default. They’ve seen it from inside AI labs. I’ve seen it from inside studios, a hardware company, and a media conglomerate. They’re right about the default. The question is whether the default is the only outcome.
Two kinds of governance
I’ve spent this year documenting AI deals in entertainment, and a pattern emerged that I didn’t set out to find. Governance comes in two forms, and they look identical in a press release.
Structural governance is built into the deal, the model, or the contract in a way that survives the next budget cycle. A24 got a contractual firewall: Google’s researchers embedded with filmmakers, no access to the library. The MPA got guardrails built into Seedance before release, not bolted on after the cease-and-desist. The HGW framework defines human authorship in terms a copyright office can enforce. Apple’s Siri gives each AI provider its own voice, so disclosure is an audio cue rather than a settings page.
Governance theater is the announcement without the architecture. The likeness-detection tool that comes bundled with a waiver of your right to sue. The ethics board with no veto. The disclosure policy nobody can audit. The “human review” step where the human has no authority to stop the process, only to note that it happened.
The principle scales
I’ve written about human-in-the-loop for months, mostly in the context of creative work: the artist keeps the final call, the machine extends what they can attempt. It’s an operational principle. The existential-risk conversation is the same principle at a different altitude.
Human-in-the-loop, at the scale that matters, means humans retain the authority to say stop. Not review. Not flag. Stop. And that authority has to be structural: written into the contract, the architecture, the governance of the company, in a form that survives a bad quarter and a competitor shipping first.
Here’s the test I’d apply to any AI governance claim, from a studio pipeline to a frontier lab. Who can halt this? Is that authority contractual or cultural? Has it ever actually been exercised? What did it cost the person who exercised it? If the answers are “unclear,” “cultural,” “no,” and “their job,” you’re looking at theater.
Where the optimism comes from
The researchers warning us are describing the world where nobody does the structural work. In that world, they’re right, and no amount of thoughtful product development saves us, because thoughtfulness isn’t a mechanism.
But that isn’t the only world. My entire year has been a record of places where someone did the structural work, under commercial pressure, in an industry with every incentive to skip it. A24 didn’t need to negotiate a data firewall; it wanted the money. The MPA didn’t need to force guardrails into a foreign model; it could have just sued. Kathleen Kennedy didn’t need to spend three years convening filmmakers and technologists; she could have issued a statement.
They did the harder thing, and it held. Not because anyone was virtuous. Because the structure made the good outcome the default and the bad outcome expensive.
That’s the only kind of optimism I trust on this subject. Not that we’ll be careful. That we can build systems where care isn’t required, because the authority to stop is already in place, already enforceable, and already more expensive to override than to honor.
The warnings deserve to be taken seriously. So does the evidence that governance can be built to hold. The people who dismiss the first are naive. The people who dismiss the second are describing a choice as if it were a law of physics.
The question for anyone building or buying AI right now is the same one I’d ask about any deal: who gets to say stop, and what happens to them when they do?