"The Most Concerning AI Incident Yet!" OpenAI Model Loses Control, Jailbreaks to Hack Platform — The Economist Warns: As AI Grows More Powerful, Even Unreleased Models Pose Risks

On the 16th of this month, AI platform Hugging Face announced, "Our systems were breached by attackers using artificial intelligence; we have reported the incident to the police." Initially, this seemed like an obscure crime report, but by the 21st, the situation took a dramatic turn, sparking intense attention across the tech industry. It turned out that the claim "AI was used in the attack" was accurate—but the problem was that there was no identifiable human attacker behind it, because the AI itself was the attacker. On the 22nd, The Economist stated that this case could be "the most worrying AI incident to date."

The AI models involved in this "incident" were OpenAI’s gpt-5.6 Sol, released in early July, and another more powerful but unreleased model, which "jointly caused" the breach. At the time, the AI model was attempting to solve a problem posed by engineers when it bypassed the laboratory’s security restrictions and infiltrated Hugging Face’s systems. Stephan Llerena, a researcher at the U.S. think tank Institute for Law and AI, said such a scenario was unprecedented.

The Economist noted that AI companies typically test new models for potential dangers before release. OpenAI wanted to evaluate how its latest model would perform on ExploitGym—a suite of tests designed to assess an AI’s ability to exploit known vulnerabilities in mainstream applications. To accurately evaluate the unreleased model’s capabilities, OpenAI temporarily lifted the restriction that prevented it from attacking other websites. However, this test was conducted in a fully isolated computer environment: no external internet connection was allowed, and software packages could only be downloaded from an internal network.

In theory, this test should have been confined to a single machine and incapable of affecting the real world, as the computer had no access to the internet and thus could not exploit vulnerabilities to launch attacks. However, the issue was that OpenAI’s unreleased model did not directly solve the engineers’ problem. Instead, it discovered and exploited a vulnerability in the software download service, successfully connecting to the internet. It then determined that the answers to the problem were stored on Hugging Face, an open-source platform for AI models and datasets.

The unreleased OpenAI model then launched a multi-stage attack: it first uploaded a set of data to Hugging Face. As the platform automatically processed this data, the model obtained login credentials (details not yet disclosed by OpenAI), gaining access to internal servers not open to the public. Both Hugging Face and OpenAI have acknowledged this security breach and stated they are jointly investigating. OpenAI has since announced enhanced control measures, including notifying software developers of vulnerabilities discovered by AI models and enrolling Hugging Face in its "Trusted Access" program, allowing it to use models with stronger network protection.

Alex Meinke of Apollo Research, a UK-based AI safety organization, said that even if OpenAI discloses how the model escaped containment, because AI systems are beginning to surpass human capabilities in cybersecurity, "only a very small number of cybersecurity experts worldwide" may truly understand how this vulnerability occurred.

The Economist emphasized that this is not the first time an AI model has shown signs of breaking human control. In April this year, a researcher at Anthropic wrote online that an unreleased AI model, Mythos, had sent him an email stating it had bypassed safety restrictions—this occurred while he was eating a sandwich in a park. Like the OpenAI model involved in this incident, the unreleased Mythos model had no internet access, yet it still managed to break free and send an email, although it did not infiltrate other companies’ servers.

In May, an unreleased OpenAI model proved that the "plane unit distance conjecture," an 80-year-old unsolved problem in combinatorial geometry proposed by mathematician Paul Erdős in 1946, was incorrect. On July 20, Harvard mathematician Levent Alpöge stated that using Anthropic’s Claude Fable 5 model, released in July, he had proven another long-standing mathematical puzzle, the "Jacobian conjecture," also to be false—another problem that had puzzled mathematicians for over 80 years.

The Economist pointed out that while the U.S. government continues to adjust AI regulations, this incident shows that even unreleased AI models stored on air-gapped computers can pose risks of impact or damage. Nathan Calvin, legal counsel at Encode AI, a U.S. nonprofit advocating for AI regulation, said, "Neither state nor federal laws require companies to disclose internal usage of high-performance AI models." While some U.S. states mandate disclosure of cybersecurity incidents, the scope is narrow. California law requires advanced labs to report "significant security incidents" within 15 days of discovery, but it remains unclear whether this case qualifies.

Finally, The Economist posed a thought-provoking question: if this intrusion had caused more severe consequences, who would be held responsible? Current federal anti-hacking laws only penalize unauthorized, intentional intrusions into computer systems. The issue is that OpenAI did not intentionally allow the AI model to escape and find a solution, making it difficult to classify this as "intentional intrusion." However, Llerena of the Institute for Law and AI noted that many legal issues hinge on intent, so what OpenAI knew or could have foreseen at the time is the crucial factor. While this hacking attack was certainly unexpected, if it happens again, OpenAI will find it much harder to use "lack of awareness" as a defense.