Philosophy

Prior reading: Competitive Dynamics and Safety Humans and AI Are Limited by Different Things Human Bottlenecks Working memory: ~7 items. Can't hold a full codebase in mind. Speed: Slow reading, slow typing, slow context-switching. Attention: Serial processor. Can focus on one thing at a time. Consistency: Fatigue, boredom, emotion. Performance degrades over a day. Bandwidth: Eyes and fingers. We interact with computers through tiny I/O channels. AI Bottlenecks Context window: Large but finite. No persistent memory across sessions without scaffolding. Grounding: No physical intuition, no lived experience, no embodied sense of consequence. Reliability: Confident and wrong. Hallucinates. Struggles with precise multi-step reasoning over long horizons. Agency: No persistent goals, no ability to autonomously decide what to work on next (without scaffolding). Verification: Can generate but struggles to verify its own outputs. These are different bottlenecks. Tools optimized for one set actively conflict with the other. ...

Prior reading: P-Hacking and Benchmarks | The Specification Problem The Setup AI writing assistants can now help researchers improve papers. Sounds good. But "improve" means "optimize for acceptance" — and acceptance is determined by reviewers with biases, time constraints, and heuristics. What the AI Optimizes Framing: Present results in the most favorable light Buzzwords: Match the vocabulary reviewers respond to Structure: Follow templates that reviewers associate with quality Claims: Calibrate confidence to what reviewers will accept without pushing back Related work: Cite the likely reviewers' papers None of this is about being more true. It's about being more accepted. ...

Prior reading: Decision Theory for AI Safety | Competitive Dynamics and Safety A version of this argument was originally posted on LessWrong. I've reworked it here with some additional context and a less formal tone. I'm a computer scientist, not an astrophysicist — corrections welcome. The Problem The Fermi paradox asks a simple question: given the size and age of the universe, where is everyone? Standard answers include: life is rare (the Great Filter), civilizations destroy themselves before going interstellar, the distances are too vast, or everyone is hiding from everyone else (the Dark Forest, from Liu Cixin's Three-Body Problem). ...

Prior reading: Game Theory for AI Safety | Decision Theory for AI Safety | Competitive Dynamics The Question If AGI is developed, what stable configurations could the world settle into? Not all equilibria are equally survivable. Possible Equilibria Many Competing Systems Multiple AGI systems operated by different actors (nations, companies). No single dominant system. Closest to today's trajectory. Stability: Moderate. Competition continues. Arms race dynamics persist. Risk of conflict, but also checks and balances. ...

Prior reading: Layers of Safety The Chinese Room Searle's argument: a person following rules to manipulate Chinese symbols doesn't understand Chinese, even if the system's outputs are indistinguishable from understanding. The component (person) lacks understanding; does the system have it? ROMs That Are Smarter Than Their Parts A read-only memory chip contains no intelligence. But a ROM storing a chess engine's entire game tree would play perfect chess. The system (ROM + lookup procedure) is "more intelligent" than its parts. ...

Prior reading: Probing: What Do Models Actually Know? The Platonic Representation Hypothesis As models scale and train on more data, their internal representations appear to converge — different architectures, different modalities, even different training objectives produce increasingly similar feature spaces. Are models discovering the "true structure" of the world? What This Looks Like Vision models and language models learn similar geometric structures Larger models have more similar representations to each other than smaller ones Cross-modal transfer works better as models scale The Platonic Analogy Plato's forms: there exist ideal, abstract representations of concepts that physical instances approximate. Near-capacity models may be approximating these forms — not because they're philosophical, but because the data constrains the geometry of good representations. ...