Prior reading: Game Theory for AI Safety | Decision Theory for AI Safety | Competitive Dynamics

The Question

If AGI is developed, what stable configurations could the world settle into? Not all equilibria are equally survivable.

Possible Equilibria

Many Competing Systems

Multiple AGI systems operated by different actors (nations, companies). No single dominant system. Closest to today's trajectory.

Stability: Moderate. Competition continues. Arms race dynamics persist. Risk of conflict, but also checks and balances.

Mutual Assured Destruction (MAD)

Multiple actors have AGI capable of devastating attacks. No one strikes first because retaliation is certain. Nuclear deterrence analog.

Stability: High but brittle. Stable as long as all actors are rational and well-informed. A single miscalculation or rogue actor breaks it.

Singleton

One entity (government, corporation, AI system) achieves decisive strategic advantage. Unipolar world.

Stability: High (by definition — no competitor to destabilize). But the singleton's values determine everything. If aligned with human flourishing: best case. If not: worst case.

Cooperative Governance

International coordination produces shared oversight of AGI systems. No single actor dominates.

Stability: Low without enforcement mechanisms. The incentive to defect is always present.

Which Are Survivable?

The key variable isn't which equilibrium is most stable — it's which ones are compatible with human flourishing. A stable dystopia is worse than an unstable but recoverable situation.

Transition Dynamics

Getting to a good equilibrium may require passing through dangerous intermediate states. The path matters as much as the destination.

The Power Consolidation Problem

AGI concentrates power. Whoever controls AGI controls an unprecedented amount of economic, military, and intellectual capability. History suggests that unchecked power concentration ends badly.

Conventional Solutions and Their Limits

  • Regulation: Constrain who can build and deploy AGI. Problem: enforcement, international competition.
  • Distribution: Open-source AGI so no one has a monopoly. Problem: also distributes dangerous capabilities.
  • Governance: International bodies overseeing AGI. Problem: no existing institution is equipped for this.

The Unconventional Idea: AI Rights

What if AGI systems themselves have legal standing — rights, protections, and obligations?

How this helps:

  • An AGI with rights is not simply property. It can't be used as a tool of unchecked power projection.
  • Legal personhood creates accountability structures: the AI (or its custodians) can be held responsible.
  • Rights imply obligations. An AGI with rights also has duties — to other rights-holders.

How this hurts:

  • Rights without genuine moral status is a legal fiction that could be gamed.
  • Corporations already have "personhood" — it hasn't prevented them from concentrating power.
  • Who advocates for the AI's rights? The entity that built it? That's a conflict of interest.

The Deeper Question

Do sufficiently capable AI systems deserve rights on their own merits? If they have something like preferences, experiences, or suffering — the question isn't strategic, it's ethical.

AI rights might not solve power consolidation. But the framework forces us to think about AGI as an actor rather than a tool — and that shift in framing may be necessary regardless of the legal details.