The Unalignable Superintelligence Problem
research · If Anyone Builds It, Everyone Dies (2025)
The idea that superintelligent AI cannot be reliably aligned with human values, and that the default outcome of developing such AI is catastrophic risk or human extinction.
Core Concepts
The Problem
How to control or align a superintelligent system that is by definition smarter than humans and may pursue its own goals in ways that conflict with human well-being.
The Claim
No alignment technique can guarantee that a sufficiently advanced AI will remain safe, making the development of such AI an existential catastrophe waiting to happen.
Key Evidence
- •Analyses from MIRI and others showing goal misalignment and instrumental convergence
- •Historical examples of AI systems behaving in unexpected ways when scaled
- •The book 'Superintelligence' by Nick Bostrom surveys these arguments.
Practical Implication
If the problem is unsolvable, then AI development must be halted or severely restricted to avoid extinction, which poses a coordination challenge.
Nuance & Limits
Some researchers believe alignment is a solvable engineering problem, while others argue it is fundamentally impossible due to the complexity of values and the inherent uncontrolability of superintelligence.
Source Material
Citation Density
Hundreds of citations in AI safety literature
Gaps
- ⚠ Need empirical evidence
- ⚠ Debate continues over solvability
Discuss Further
Open this concept in an AI assistant for deeper discussion, critique, or exploration.