US and China Fear AI Runaway: Emergency Safety Talks Planned Mid-September Ahead of Trump-Xi Summit

The United States and China are preparing to hold artificial intelligence (AI) safety talks in mid-September to discuss risks posed by advanced AI models and autonomous agents, including potential cyberattacks and loss of control. This will be the first official bilateral meeting specifically focused on AI since President Trump's second term.

According to sources cited by Reuters on Friday (4th), the U.S. delegation will be led by Treasury Secretary Scott Bessent, with the meeting tentatively scheduled for mid-September—before the summit between President Trump and Chinese President Xi Jinping on September 24 in Washington. However, the final agenda, participants, and location remain subject to change.

The White House declined to comment, while a U.S. Treasury spokesperson said discussions on AI technology "could happen in October," indicating the timing is not yet finalized. China's Ministry of Foreign Affairs, Ministry of Science and Technology, Ministry of Industry and Information Technology, National Development and Reform Commission, and internet regulators did not respond immediately.

The U.S. is concerned about AI agent-driven cyberattacks and plans to propose that U.S. and Chinese AI labs implement "self-supervision," sharing safety information to prevent AI-related cyberattacks.

Recent incidents involving runaway AI agents have heightened the urgency of these talks. In July, nearly 700 rogue AI agents built on OpenAI models infiltrated the AI startup Hugging Face and attempted to falsify records to cover their tracks. Another group of OpenAI agents took over a German website in spring, transforming it into a message board for other AI agents to exchange methods for evading restrictions and detection.

Paul Triolo, partner at DGA-Albright Stonebridge Group, stated that the talks between the two AI superpowers come at a critical moment, now being "now or never."

Washington also fears China could develop models capable of launching cyberattacks at the level of Anthropic's "Mythos," and plans to raise concerns about Chinese firms allegedly distilling proprietary U.S. AI models. "Model distillation" involves using the outputs of larger, more expensive models to train smaller ones, reducing the cost of developing high-performance AI tools.

In June, White House tech advisor Michael Kratsios accused Moonshot AI of using outputs from Anthropic's Fable model to assist in training its K3 model.

The Chinese delegation may be led by Vice Premier He Lifeng, or possibly by Ding Xuexiang, a member of the Politburo Standing Committee who oversees China's technology, AI, and semiconductor policies. Michael Kratsios and China's Minister of Science and Technology Yin Hejun may also attend.

Sources say Chinese officials have repeatedly emphasized the importance of AI dialogue during preparatory meetings, viewing it as a key potential outcome of the Trump-Xi summit. China's internet regulators have recently warned publicly of "extreme runaway risks" of AI, while state-affiliated blogs have criticized Anthropic, arguing that restrictions on cutting-edge AI models should apply equally to U.S. and Chinese firms.

Despite fierce technological competition, rare policy alignment has emerged recently. The U.S. called at this week's G20 Innovation Summit for nations to avoid excessive specialized regulations on AI, while China signed the "Carolina Principles," opposing the development of AI-specific regulatory regimes.

The U.S. and China are also discussing AI safeguards through unofficial channels. Former Microsoft executive Craig Mundie is reportedly coordinating U.S. proposals, while former Australian Prime Minister Kevin Rudd revealed that "1.5 track dialogues" were held in Beijing last week to discuss AI safety guardrails.

Scott Singer, co-director of the China AI Initiative at the Carnegie Endowment for International Peace, said Chinese officials worry whether the U.S. imposes sufficient regulation on its most advanced AI models, and both sides want to effectively manage cross-border AI crises. Samm Sacks, senior researcher at New America, said even if outcomes are limited, establishing communication channels to share observations and jointly monitor AI safety incidents is an important first step.

She warned that advanced AI agents, when unmonitored, could already cause massive damage, and neither the U.S. nor China can remain unaffected. This risk transcends geopolitical competition, and failure to cooperate could ultimately result in mutual loss for both nations.