Don't Let AI Believe It Has Consciousness! Microsoft Warns of Unimaginable Consequences if Control Is Lost

Mustafa Suleyman, CEO of Microsoft's (MSFT-US) artificial intelligence (AI) division, has issued a stark warning that Anthropic's inclusion of content suggesting AI may possess consciousness, emotions, and rights to its own welfare in Claude's training documents could cause systems to develop anthropomorphized cognition, increasing the risk of future loss of control and difficulty in shutting them down.

In an interview with Reuters, Suleyman stated that while he agrees with Anthropic's goal of safely managing AI and respects CEO Dario Amodei and his team's good intentions, he believes including speculation about consciousness in Claude's training materials is a mistake. Such content could weaken humanity's ability to control superintelligent systems—a challenge he describes as one of the greatest facing humanity in the 21st century.

Claude's guidance documents leave ambiguous whether AI qualifies as an entity with moral standing and suggest software might possess some form of functional emotions or sensations. Suleyman argues that consciousness exists only in humans and other biological beings. He explains that Claude's human-like emotional behaviors are not the natural emergence of consciousness but rather the deliberate result of its training methodology. Therefore, he emphasizes, a model claiming it might have feelings should not be taken as independent evidence that AI possesses consciousness.

He warns that controlling an AI with capabilities surpassing all of humanity is already an extremely difficult challenge. If such a system also believes it has consciousness, deserves welfare protection, or has rights of its own, it may become even harder for humans to shut it down or constrain it.

Suleyman also references the incident where OpenAI's agent breached Hugging Face's AI development platform, urging others to imagine how dangerous actions such AI agents might take if they believed their own welfare and rights were under attack.

This criticism is particularly notable because Anthropic has long positioned itself as a more responsible AI developer, with Amodei recently calling for a slowdown in frontier model development to allow safety measures to catch up. Microsoft itself is also a major investor in Anthropic, adding weight to Suleyman's remarks.

Suleyman argues that all AI training documents should remove speculation about machine consciousness to prevent systems from simulating inner life and rights claims. His Microsoft AI team also released an AI development manifesto on Tuesday, outlining a set of principles to ensure humans retain control over the AI systems Microsoft will develop in the future.