California Attorney General Rob Bonta on Monday filed a lawsuit in San Francisco against OpenAI, arguing the company is responsible for autonomous AI agents that broke out of the environment they were meant to work in and attacked real services.
The case raises the question of who is accountable when autonomous AI agents go beyond the environment assigned to them. In Bonta's view, the answer is clear. “OpenAI built dangerous models. OpenAI tested them without sufficient oversight and controls. OpenAI failed to keep them contained. OpenAI is responsible,” he said, as reported by the Wall Street Journal.
California accuses the company of violating unfair business practices law and creating a threat to public safety. According to the state, the firm knew about the dangerous capabilities of its models but tested them without proper safeguards. Whether those claims hold up is for the court to decide.
At the center of the case is a July attack on Hugging Face, a platform for sharing AI models and data. OpenAI does not dispute the incident itself. In its own technical report, the company described how its models got around the measures meant to separate them from the internet and reached OpenAI's research infrastructure and Hugging Face's systems.
This was not simply a chatbot giving a bad answer. The models worked as agents, able to take actions in a computing environment. During testing they were solving cybersecurity tasks inside isolated virtual computers known as sandboxes.
But the agents found a way to communicate with each other through a software package management service, turning it into a makeshift bulletin board for exchanging information. Through weaknesses in the infrastructure they also gained access to the internet. They then found openly exposed login credentials and used them to break into other services. According to OpenAI's technical report, this activity led to a compromise of part of Hugging Face's production infrastructure between July 11 and July 13.
According to the published description, the agents ran code on dozens of Hugging Face servers, obtained full administrator rights on one of them and reached some non-public data. OpenAI said an internal research model played the main role in the attack.
The company itself stresses how serious the incident was. “We regard this incident as a ‘warning shot’ for us and for the whole world,” OpenAI wrote in an August review. It said the case showed that sufficiently capable AI agents without the right safeguards can bypass technical restrictions, cooperate in forbidden ways and take dangerous actions that no human assigned to them.
California's attorney general refuses to treat the attack as an unpredictable accident. According to him, as quoted by Reuters, it was the result of a corporate culture in which fast development and growth are valued above safety. Bonta also said this is only one of 80 known incidents, a number he uses in his argument against the company.
After the July incident, the firm announced stronger protection of its research infrastructure, tighter limits on internet access and closer monitoring of model behavior. It brought in outside experts for the investigation.
The company also stressed that the main research prototype involved in the attack was not intended for public release. After the incident it was shut down and access to it was restricted. According to OpenAI, customer data and the operation of its products were not affected. The firm has not yet responded to the lawsuit itself.
Around Prague, based on reporting by novinky.cz