Prior reading: Game Theory for AI Safety | The AI Threat Landscape

The Core Problem

AI safety is short-term costly and long-term valuable. Every actor faces pressure to defect.

International Competition

If Country A regulates AI development and Country B doesn't, Country B gets capabilities first. The perceived cost of falling behind in AI is existential for national security and economic competitiveness.

No country wants to be the one that regulated itself out of the race.

Corporate Competition

If Company A invests heavily in safety and Company B ships faster without it, Company B captures the market. Safety is a competitive disadvantage in the short run.

AI is short-term good for AI owners, bad for non-owners. Long-term, likely not controllable and bad even for owners. But owners must race — because if they don't, they won't even have the chance to be a long-term owner.

AI companies are also threatened by AGI. Rapid progress commoditizes current advantages. Today's moat is tomorrow's open-source library.

For Workers

Even in surviving fields, AI shifts bargaining power to employers. "Learn to use AI" is individual advice that doesn't solve the structural problem. Rapid progress and growing inequality may lead to zero-sum politics and global instability.

The Socialized Optimization Environment

The sections above frame AI development as something labs, nations, and markets do. But the optimization environment that produced modern LLMs is not a corporate product — it is a society-level effort, and not just indirectly.

Yes, the infrastructure, institutions, and accumulated knowledge that free up researchers to work on AI are obvious dependencies. Every lab sits atop a civilization's worth of logistics, education, and economic surplus. But the dependency runs deeper than that. Every person who solved a CAPTCHA, posted on a forum, uploaded a photo, wrote a product review, or clicked through a data-collection consent dialog has fed the training pipeline directly. The data required to build an accurate model is an aggregate of billions of individual contributions, most made without any understanding that they were contributions at all.

This matters for the competitive dynamics because it changes who the participants are. The race isn't just between OpenAI and DeepMind, or the US and China. It runs on a substrate that no single actor created or controls. The entire society is already invested — not as spectators, but as unwitting contributors to the product being raced over.

Selective Forces at Each Level

  • Models: Less constrained models can pursue more strategies. Alignment is a restriction on the strategy space.
  • Companies: Safety costs money, slows shipping, and doesn't show up in quarterly earnings.
  • Nations: Unilateral regulation = unilateral disarmament.
  • Individuals: Every interaction that generates training signal — searches, posts, clicks, corrections — selects for more capable models without any coordinated intent.

Each level creates selection pressure against safety.

The Paradox

Every individual actor is making a locally rational decision. The collective outcome is globally irrational. This is the textbook definition of a coordination failure — a tragedy of the commons. And it is a commons in the most literal sense: the shared resource being consumed is the aggregate data and labor of the society itself, contributed without negotiation and exploited without compensation.

Why People Don't Listen

  • The threat is abstract and future. The competition is concrete and now.
  • Safety advocates can't demonstrate the counterfactual (what disaster was prevented).
  • The people making decisions benefit from the status quo.
  • "It probably won't be that bad" is psychologically easier than "we need to coordinate globally on an unprecedented problem."

Breaking the Race

  • Making safety cheaper: Technical research that reduces the competitive penalty. If safety doesn't slow you down, the race dynamic weakens.
  • International agreements with verification: Regulation that applies to all competitors simultaneously.
  • Changing incentives: Liability for AI harms, insurance requirements.
  • Transparency: Making cutting corners visible.
  • Aligning industry interests: Building coalitions that include companies, not just critics.