A former researcher at Anthropic and OpenAI, Jacob Coxon, warned that leading AI firms are pressing ahead despite catastrophic risks, estimating a double-digit probability that advanced systems could pose an existential threat within a decade. His departure and public critique drew rare, swift affirmation from other researchers, including at Anthropic, underscoring rising concern over recursive self-improvement and recent incidents involving AI-enabled intrusions. While some critics see talk of “doomer” scenarios as a bid for regulatory capture, Coxon argues the private sector cannot self-police amid competitive and geopolitical pressures, particularly the race with China. With no comprehensive U.S. rules in place, momentum is building for pre-release testing and greater oversight, but there is little consensus on what regulation should look like. Coxon and allies call for global coordination and potential slowdowns if models surpass safety thresholds—an objective that remains politically remote as capability advances accelerate.
Related articles:
The Bletchley Declaration on AI Safety
NIST AI Risk Management Framework
