Preventing AI Autonomous Upgrade Risks! OpenAI Proposes New Safety Standards Focusing on Alignment Research and RSI Technology Risks

On Monday (21st), OpenAI published a set of recommendations regarding the safety and safeguards for cutting-edge artificial intelligence (AI) development, focusing on 'alignment research' and a computational technique known as 'recursive self-improvement' (RSI).

In a blog post, OpenAI stated: 'To safely navigate this transformation, alignment research must keep pace with the development of these capabilities, ensuring that the systems we and other developers build remain aligned with human values and under human control.'

OpenAI is urging countries to engage in international cooperation to jointly establish advanced AI standards, suggesting building upon the work already being done by existing global AI safety research institutions.

OpenAI says these technical standards should focus on frontier AI models and developers, as well as managing the benefits and risks of automating AI researchers—including RSI.

RSI has drawn attention from AI developers due to its potential to create foundational models that allow AI systems to upgrade themselves without human intervention.

However, progress in the field of RSI has also raised concerns among some technologists, who worry that model developers might lose control over the core technology, or fail to adequately consider unintended consequences as AI systems grow increasingly complex and widely distributed across the internet.

OpenAI wrote in its blog post: 'Fully autonomous RSI has not yet been achieved, and we should not pursue this technology unless and until it can be realized in a safe manner. Without proper attention and caution, RSI could lead to humans losing practical control over AI development, and even prevent us from supervising research processes we can no longer understand.'

OpenAI’s article also referenced the incident where Hugging Face was breached by an AI agent. While this event did not involve RSI technology, OpenAI views it as a rehearsal for how related risks could dramatically intensify in the future if robust protective measures and alignment mechanisms are lacking.

Competitor Anthropic also unveiled concepts last week for the safe development of frontier AI models, responding to recent warnings from industry researchers about AI potentially threatening humanity.

Jacob Coxon, formerly employed at both Anthropic and OpenAI, announced his resignation about two weeks ago, stating at the time that these companies were 'betting with our lives,' sparking global discussion. Following a series of recent AI-related security incidents and Coxon’s public remarks, Anthropic CEO Dario Amodei published an article urging AI companies to slow down the development of foundational models and proposing additional safety measures.

Amodei also suggested allowing third-party assessment organizations to enter AI companies to review and mitigate societal risks posed by the technology, such as accelerating cyberattacks or aiding in the production of biological weapons.

Sam Altman, CEO of OpenAI, and Elon Musk, CEO of SpaceX, among other rival leaders, have publicly supported Amodei’s proposal.

However, because the field of AI evaluation remains in a very early stage, there is currently no unified consensus, nor any set of basic standards and principles that would allow independent third parties to conduct deeper technical audits and inspections than they can today.

This is partly why the Coalition of AI Evaluation Organizations is calling on foundational model developers to consider a 'minimum bar'—enabling evaluators to perform more in-depth technical audits and checks, including gaining deeper system access and avoiding retaliation for publishing unfavorable reports.