V. The Prep Window

If prep costs pennies but failure could cost everything, why wait? From a game-theoretic view, early AGI prep is the dominant strategy. The core logic is Pascal's wager: preparing early costs little; being unprepared could be catastrophic. Societies often sleepwalk into predictable crises—as evidenced by COVID-19. With AGI, the stakes are existential. Uncertainty demands preparation, not inaction.

The art of war teaches us to rely not on the likelihood of the enemy's not coming, but on our own readiness to receive him; not on the chance of his not attacking, but rather on the fact that we have made our position unassailable.

— Sun Tzu

To be prepared for war is one of the most effective means of preserving peace.

— George Washington

If you’re enjoying this deep dive and want to keep up to date with my future writing projects and ideas, subscribe here.

The storm's coming

Humans show a remarkable capacity to recognize approaching danger while simultaneously failing to act on that knowledge. This pattern emerges from a complex interplay of structural and psychological factors: the tragedy of commons in distributed responsibility, short-term political and corporate incentives, and our natural discomfort with contemplating existential threats.

Throughout history, we've consistently assumed "someone else" will handle looming crises—from ancient wars to modern pandemics. Yet with artificial general intelligence (AGI), this tendency toward collective abdication faces its most consequential test. Unlike previous technologies that developed through visible, gradual stages, AGI follows an exponential progression that might compress decades of advancement into months once certain thresholds are crossed.

The wisdom of strategy teaches us a simple truth: if you want to survive a storm, you don't wait until lightning strikes. You prepare while the sky is still blue. With AGI, we're still under mostly blue skies, but clouds are gathering on the horizon—and history suggests¹ they move faster than people expect. AI could be the fastest storm of them all.

The most important lesson from strategy, finance, and even good old common sense: it's never enough to merely bet that disaster won't happen. You must build a position strong enough to survive if it does.

1 The one-way gamble

Pascal’s wager is one of the oldest arguments for why it’s rational to prepare for uncertain but high-stakes events. The original version was about God. Blaise Pascal, the seventeenth-century philosopher and mathematician, argued that you should live as if God exists, because if you're wrong, you lose a little — but if you're right, you gain everything. In his words: if God doesn't exist, you lose only finite pleasures. But if He does, you gain eternity — and avoid eternal loss².

Pascal contends that a rational person should adopt a lifestyle consistent with the existence of God and actively strive to believe in God.

The reasoning behind this stance lies in the potential outcomes: if God does not exist, the individual incurs only finite losses, potentially sacrificing certain pleasures and luxuries.

However, if God does indeed exist, they stand to gain immeasurably, as represented for example by an eternity in Heaven in Abrahamic tradition, while simultaneously avoiding boundless losses associated with an eternity in Hell.

It’s not a perfect argument for questions of faith — people like Richard Dawkins have pointed out the problems. But it’s still a useful way³ to think about risks where the downside is massive and the upside of preparation is much larger than the cost.

You see this logic in action with climate change. Even if you thought there was only a 1% chance it’s real, the scale of possible disaster means you have to take it seriously. Not preparing is bad math⁴.

The same thing happened with COVID-19. If you were willing to entertain the possibility early — even if you weren’t sure — you could take simple steps to protect your family, move if needed, stock up, or avoid getting trapped somewhere bad. You didn’t have to be certain to act. And acting preserved your agency when the crisis hit⁵.

If you ignored the possibility altogether, you lost your ability to choose. You ended up at the mercy of the system around you. It’s the same with AGI. We can’t know for sure if we’ll get it in five years. But if we do, the world will change faster than anything we’ve ever seen. The real question is simple: If that future comes, do you want to be prepared — or caught flat-footed?

2 Asleep at the edge

Fool me once, shame on you; fool me twice, shame on me.

— Anthony Welden

One reason I take AGI seriously—and on a short timeline—is because history repeats. Our collective capacity for self-deception in the face of visible catastrophe is a persistent feature of history. Just ask Winston Churchill. He warned of the threat of Hitler ad nauseam before he was taken seriously and eventually awarded the prime minister role—luckily, it wasn't too late.

Consider how almost everyone sleepwalked into COVID-19, despite clear warnings. You might argue the death toll wasn't catastrophic compared to historical plagues, but that was more fortune than foresight. And AGI, if it goes wrong, won't just affect some people—it will touch everything and everyone.

The pandemic provides direct evidence that even when risks are obvious and experts are raising alarms, most of the world stays focused on daily concerns while ignoring the looming crisis. The warnings were there, accumulating over years⁶:

In 2015, Bill Gates warned in his TED Talk that our next catastrophe would likely be a virus, not war, and that our response systems were inadequate. By 2017, epidemiologist Michael Osterholm had published Deadliest Enemy, declaring a pandemic inevitable and outlining preparation plans that governments largely ignored. That same year, Jeremy Konyndyk predicted in Politico that U.S. neglect of health emergency systems would lead to disaster. By 2018, Ed Yong wrote plainly in The Atlantic that global travel, urbanization, and weak health systems made a deadly pandemic inevitable, stating "much worse is coming." Most telling, just months before COVID-19 struck, the Johns Hopkins Center for Health Security reported that not a single country—not even the wealthiest—was fully prepared for a pandemic.

And yet when the pandemic arrived, almost no one was ready.

If that's how the world reacts to a threat it can see coming, what makes us think it will be different with AGI? The pattern seems disturbingly consistent: experts warn, institutions acknowledge but fail to act, and preparation remains theoretical until crisis forces a response.

This time, even more voices are sounding the alarm about artificial intelligence. This time, we've been warned. The question is: have we learned to listen?

3 The only certainty is uncertainty

Our previous analysis suggests three possible timelines: AGI around 2030 appears most plausible; timelines beyond five years remain moderately plausible; while development within the next five years seems least plausible, though we can't dismiss it entirely.

The uncertainty stems from three opposing forces:

1. Recursive Self-Improvement⁷. AI designing better AI could accelerate progress unpredictably, shortening timelines.
2. Constraints⁸. New bottlenecks (e.g., energy, data quality) might emerge, lengthening timelines.
3. Emergence⁹. Scaling could unlock sudden capabilities, shortening timelines.

These forces push in opposite directions. And no one knows which will dominate.

You can't prove we’ll get AGI in five years. You can't prove we won’t. You can't even prove it won’t happen next year¹⁰. In investing, when you face that kind of uncertainty, you don’t just guess better — you build a margin of safety¹¹. You prepare for things to go wrong¹², even if you hope they won't. That’s how you survive.

It was true in 2008 when bad forecasts sank entire companies. It will be true again with AGI. The difference between survival and disaster won’t be prediction accuracy. It will be preparation.

You don’t have to know exactly when the future will arrive. You just have to be ready when it does.

Preparation

The future isn't something that happens to you. It's something you either shape or get shaped by. We don't know exactly when AGI will arrive, or exactly what it will look like. But uncertainty isn't a reason to do nothing. It's the best reason to act.

When things change this fast, the cost of being unprepared is measured not just in money or opportunity, but in agency — in whether you're someone who gets to make choices, or someone who just has to live with them.

So the question isn’t, "Will this happen soon enough to matter?" The question is, "When it does, what position do I want to be in?" The smartest way to prepare for an uncertain future is to start early.

Coming up next: Which specific moves flip uncertainty into advantage during the transition?

Continue reading →

Want me to send you new ideas?

If you’re enjoying this deep dive and want to keep up to date with my future writing projects and ideas, subscribe here:

Subscribe to future updates →

Footnotes

1

One of AI's founders thinks we're building the equivalent of nuclear reactors without control rods. Russell's warning comes after 40 years advancing the very field he now fears.

2

Found posthumously in his papers. The probability pioneer died at 39.

3

Dawkins meant to demolish Pascal's wager but accidentally strengthened its application to extinction risks. His book raises the perfect question for AI alignment. why would a superintelligence value what humans value?

4

Standard economic models can't handle genuine uncertainty. Weitzman showed that unlikely but extreme outcomes should dominate our calculations—we've been drastically underestimating climate risks.

5

The trader who made millions betting against conventional wisdom. Systems that gain from disorder—what Taleb calls "antifragile"—might be our best defense against unpredictable catastrophes.

6

Seventeen years. That's how long we had to prepare for COVID after the first detailed warnings. This compilation documents 25 specific pandemic alerts experts ignored—including Bill Gates' now-famous 2015 TED talk.

7

The AI equivalent of compound interest—systems improving themselves, creating better systems, which improve themselves further. Bostrom coined "recursive self-improvement" to describe this potentially explosive feedback loop.

8

After two years analyzing AI progress at Open Philanthropy, Cotra gives a 10% chance of AGI by 2025 and a median estimate of 2040. Her forecast model remains the most comprehensive available.

9

Neural networks suddenly "getting it" after appearing to merely memorize data. This phenomenon, called "grokking," might be the most important AI behavior you've never heard of.

10

Either we go extinct soon, colonize the stars, or live in a simulation—there are no boring futures left according to Karnofsky's analysis.

11

The principle that made Warren Buffett the world's greatest investor. you don't need to know something's exact value—just that it's worth significantly more than you're paying. Graham's approach, applied to existential risks, means preparing for worst-case scenarios even without precise probabilities.

12

We're wired to ignore low-probability, high-impact events until they happen. Taleb predicted the 2008 crash using this insight about human blindspots.

⁶