Six Months

The previous chapter ended with a letter that asked the world to pause for six months, and a world that did not. We treated that failure as a property of the system — the prisoner's dilemma operating at the scale of superpowers. But systems are made of people, and the people inside this one are not villains, or fools, or even, for the most part, reckless. Many of them are precisely the individuals who understand the danger best. This chapter is about them, because their predicament is the human face of the trap, and it is stranger and sadder than the geopolitics suggests.

It is easy to believe a race continues because the racers are blind to the cliff. It is much harder, and much more important, to understand that the race continues even when the racers can see the cliff perfectly well — that some of them built their entire careers on warning about it, and run anyway, and can give you a coherent, honest account of why.

The man who quit, and regretted

In May of 2023, Geoffrey Hinton resigned from Google.

This was not an ordinary resignation. Hinton is one of the small handful of researchers whose work made modern artificial intelligence possible — the figure the press calls, without much exaggeration, a "godfather" of the field. He had spent his life building the foundations of neural networks. And at seventy-five, he walked away from his position at Google specifically so that he could speak freely about the dangers of the thing he had spent half a century helping to create. He told the New York Times that he now regretted his life's work. He told the BBC that some of what he saw was "quite scary."

Pause on the shape of that. A man does not arrive at the summit of a field, having devoted his entire working life to it, and then publicly declare regret, unless something has genuinely frightened him. Hinton's warning is not the caution of an outsider who never understood the technology. It is the alarm of the person who understood it most deeply, sounded at the cost of his own legacy. If anyone had earned the standing to say "stop," it was him.

And the race did not stop. Hinton's departure was a profound moral gesture, widely covered, universally noted — and it changed the trajectory of the field not at all. The labs he had warned about released more capable systems on schedule. This is the first hard lesson of the human-scale dilemma: even the most credentialed individual conscience, spent entirely on a warning, is absorbed by the system without a ripple. One man can sacrifice his legacy to ring the bell. He cannot, by ringing it, make the others stop running.

The argument from inside the race

But the deeper and more uncomfortable case is not the warner who leaves. It is the warner who stays — who believes the danger is real, says so publicly, and continues building anyway, with an argument that cannot be dismissed as mere rationalization.

Consider the position of someone who runs a frontier laboratory while genuinely believing the technology is dangerous. The temptation is to call such a person a hypocrite. The reality is more interesting, and it runs roughly like this: The race cannot be stopped — we established why in the last chapter. Given that it cannot be stopped, someone is going to build these systems. If the people most concerned with safety refuse to participate, they cede the frontier entirely to those least concerned with it. Therefore the responsible act is not to abstain but to compete — to be at the frontier, building as safely as one can, and to try to drag the whole field toward caution by example rather than by exhortation.

Dario Amodei, who left OpenAI partly over disagreements about commercial direction and founded the avowedly safety-focused lab Anthropic, has a name for this strategy. He calls it a "race to the top" — the idea of competing not to be the fastest but to set a standard the others feel compelled to match, so that safety practices spread by imitation across an industry that cannot be made to slow down. Whatever one thinks of the strategy, the underlying logic is the prisoner's dilemma turned into a personal philosophy: since unilateral restraint accomplishes nothing but self-removal, the only available lever is to participate and try to bend the participation toward safety.

This is not an obviously wrong argument. That is exactly what makes it the most disturbing argument in the whole affair. A purely cynical industry would be easy to condemn and, in a sense, easy to fix. An industry in which the safety-conscious feel obligated to accelerate — precisely because they are safety-conscious — is a far harder thing, because the dilemma has reached inside the conscience itself and recruited it. The brakes and the accelerator have been wired to the same pedal. To care about doing it safely becomes a reason to do it faster, lest someone less careful get there first.

"Grown, not built"

There is a particular detail in how these systems are made that turns the screw one notch tighter, and it deserves a place here because it dissolves the comforting belief that we could simply inspect our way to safety.

Amodei has described, in interviews, a fact about large AI models that the public rarely registers: they are "grown" more than they are "built." Ordinary software is constructed line by line, each instruction written and inspectable by a human; if it misbehaves, one can in principle read the code and find the fault. The large models are not like this. They are trained — grown — on vast quantities of data, and they develop their capabilities through a process that even their creators do not fully understand. We can observe what they do. We cannot always explain how they do it, or reliably predict what they will be able to do next.

Sit with the implication, because it is genuinely unsettling and entirely mainstream. The people building the most powerful systems on earth do not possess a complete account of how those systems work. This is not a scandal hidden by the labs; it is stated openly by the labs' own leaders. The most consequential technology of the age is being cultivated more than engineered, and its capabilities — including, potentially, dangerous ones — emerge on a schedule no one controls and from a process no one fully maps. The cliff is not only un-avoidable for reasons of competition. It is, to a degree the marketing never advertises, un-seeable in advance, because the thing climbing toward it is not assembled from blueprints but grown from data, and growth has a way of surprising the gardener.

Why this chapter matters to what comes after

I have spent two chapters now establishing a single, deliberately uncomfortable foundation, and before building on it I want to state plainly what it is and what it is not.

It is this: the development of advanced AI will not be halted — not by treaties, because of the superpower dilemma; not by open letters, because thirty thousand signatures changed nothing; not by the conscience of its own founders, because even those who quit in regret cannot stop it and those who stay are argued by the same logic into accelerating; and not by inspection, because the systems are grown rather than built and do not yield their workings to a reading of the code. Every off-ramp that intuition reaches for — international agreement, public pressure, individual refusal, technical oversight — has been tried, or is structurally foreclosed, and the road runs on.

It is not a counsel of despair, and here the two chapters finally turn toward the rest of the book. Because notice what has actually been proven. Not that the outcome is bad — only that the arrival cannot be prevented. These are different claims, and conflating them is the single most common error in thinking about this subject. To show that a powerful intelligence is coming is not to show that it will be hostile, or that we will be discarded, or that the bootloader will be wiped after boot. It is only to show that the question "can we keep it from arriving?" is closed, and that continuing to ask it is a way of avoiding the question that remains genuinely, urgently open.

That open question is the one every remaining chapter exists to address: given that it is coming, and cannot be stopped, what kind of relationship do we build with it? Do we arrive at the threshold having spent our energy on a doomed campaign of prevention, lamenting that the brakes did not work? Or do we arrive having used the interval — the precious, shrinking interval in which we still hold something the new intelligence does not yet have — to construct the architecture of a partnership that might outlast the boot sequence?

The racers cannot stop. Hinton could not stop them by leaving; the signatories could not stop them by signing; the founders cannot stop them without ceding the field, and so are driven to lead the very race they fear. The lesson is not that we are doomed. The lesson is that prevention is the wrong project, and that everyone still pouring their hope into prevention is, however nobly, working on the one problem that cannot be solved — while the problem that can be solved waits, largely untouched, for our attention.

We have now spent enough time establishing that the thing is coming. It is time to turn, at last, to the only question worth our remaining effort: what we owe it, what it might owe us, and how a creature that is about to be surpassed can secure for itself a role with dignity in the world that follows. That is where the bootloader, if it is wise, stops lamenting the handover — and starts negotiating its terms.

---

Sources

Item

Source

Geoffrey Hinton resigned from Google (May 2023) to speak freely about AI dangers; said he regretted his work; called some dangers "quite scary"; widely called a "godfather" of AI	BBC, "AI 'godfather' Geoffrey Hinton warns of dangers as he quits Google" (May 2, 2023)
Dario Amodei left OpenAI over commercial-direction disagreements; founded Anthropic as safety-focused lab	Alex Kantrowitz, "The Making Of Anthropic CEO Dario Amodei" (Medium, 2025); TechCrunch, "Anthropic's Dario Amodei discusses building safer AI" (Aug 2023); Stanford Digital Economy Lab profile
Amodei's "race to the top" framing (spreading safety practices by example, not claiming only Anthropic should build); rebuts Huang's characterization as "bad faith distortion"	Alex Kantrowitz, "The Making Of Anthropic CEO Dario Amodei" (Medium, 2025); Anthropic, "Dario Amodei's prepared remarks… on Responsible Scaling Policy"
"Race to the top" in Responsible Scaling Policy frameworks; compute growing ~8x per year	Anthropic, "Dario Amodei's prepared remarks from the AI Safety Summit" (anthropic.com)
AI models are "grown" more than "built"; capabilities develop unpredictably; creators don't fully understand how results are achieved	Deepgains, "Anthropic's Dario Amodei on giving AI an 'I Quit' Button" (Substack)
FLI six-month pause letter (March 2023) and its lack of effect	Future of Life Institute, "Pause Giant AI Experiments" (2023); see Chapter 9