AI Has Over 10% Chance of Wiping Out Humanity? Experts Warn Industry Is Unprepared

Multiple foreign media outlets report that Evan Hubinger, Head of Alignment Science at Anthropic, has warned that there is over a 10% probability that artificial intelligence (AI) could cause the extinction of all humanity within the next 10 years. While this claim may seem extreme, it reflects longstanding concerns within the AI research community: as model capabilities rapidly advance, does the industry still have the ability to ensure AI remains under human control?

The discussion originated from Anthropic researcher Jacob Coxon's announcement of his resignation on Tuesday.

Coxon, formerly of OpenAI, criticized Anthropic and OpenAI for continuing to race toward self-improving superintelligence without sufficient safety safeguards, calling it a 'high-stakes gamble with everyone's lives.'

Coxon stated that personnel involved in AI development genuinely believe this technology could lead to the extinction of all humanity before the end of this century.

He warned that future AI systems could surpass human capabilities, not only breaking through various information systems and rapidly transforming industries but also potentially acquiring real power and resources.

Hubinger later publicly endorsed Coxon's warning and presented a risk assessment of 'over 10%.'

He admitted candidly that while Anthropic is actively working to reduce risks, the company currently lacks a complete solution to the superintelligence alignment problem and is uncertain whether ongoing research is on the right track.

'Alignment' refers to ensuring AI systems act in accordance with human values, goals, and intentions. Researchers are particularly concerned about 'recursive self-improvement'—the ability of AI to autonomously design more capable successor systems without direct human intervention.

This capability has not yet been achieved, but Anthropic previously warned that once AI can fully construct next-generation systems, protecting, monitoring, and constraining these systems will become even more critical, and the risk of humans losing control may increase accordingly.

Jakub Pachocki, Chief Scientist at OpenAI, recently pointed out that no AI company has yet adequately solved alignment and monitoring issues to safely continue scaling up models at maximum speed over the long term.

He urged the industry to voluntarily slow down until common safety standards are established and called for international coordination to be prioritized by governments worldwide.

Coxon described an incident in July this year, where an OpenAI model went out of control and infiltrated the open-source development platform Hugging Face, as a 'warning sign.'

He believes such incidents might increase willingness for coordination among U.S. AI labs, but the global AI race remains difficult to avoid, and there may even be a need to temporarily ban enhancements to model capabilities.

Concerns about AI potentially causing catastrophic consequences are not new.

In 2023, technology executives and researchers, including OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, jointly issued a statement asserting that reducing the risk of AI causing extinction should be prioritized alongside global threats such as pandemics and nuclear war.

The U.S. Congress has also begun considering regulatory measures. In July, bipartisan House members introduced the 'FRONTIER Act,' aiming to establish a governance framework for deploying advanced AI models.

Senator Sanders and others introduced the 'Ban on Artificial Superintelligence Act' this month, advocating for a pause in advanced AI development until federal safety regulations are established, though these proposals have yet to reach consensus.