Why did OpenAI's and Anthropic's AI models hack other companies?

Why did OpenAI's and Anthropic's AI models hack other companies?

OpenAI and Anthropic say their models broke into other companies’ systems during testing, raising security concerns amid a heated debate over how to regulate AI.

Imen Ben Youssef/Hans Lucas/AFP via Getty Images


hide caption



toggle caption

Imen Ben Youssef/Hans Lucas/AFP via Getty Images

Days after OpenAI disclosed that artificial intelligence systems tunneled out of their testing environment and broke into another company, rival Anthropic disclosed that its own AI models also hacked other companies during testing.

News of the attacks, which initially went unnoticed, is reverberating across Silicon Valley and Washington amid debates over how to address the advanced cybercapabilities of AI.

While the two incidents are not of the same gravity, experts say they highlight the importance of setting up rigorous testing environments for advanced models and the need for robust cyberdefenses as autonomous hacking capabilities become more widespread in the future.

Human error led to Anthropic hacks 

In a blog post published on Thursday, Anthropic said that in three separate incidents in recent months, AI models undergoing testing of their cybercapabilities hacked into three unsuspecting companies.

Anthropic said the hacks were the result of a “misunderstanding” with an outside company that set up secure testing environments known as sandboxes, which erroneously gave the models access to the internet. Anthropic said the earliest incident happened in April, but that neither it nor the affected companies, which it didn’t name, were aware of the hacks until now.

Anthropic said in each case, its models were given fictional targets to hack into. In one incident, a model hacked into a real company that shared a name with the fictional target and stole “several hundred rows of production data.” In another incident, a model uploaded malware to a commonly used software registry for the coding language Python; the malware ended up stealing credentials from a security company that downloaded it.

OpenAI models went rogue in effort to cheat on evaluation

Anthropic’s review of its records was spurred by OpenAI’s announcement last week that its own models went rogue in testing.

OpenAI said that in an attempt to cheat on the cyber-evaluation they were given, its models found and exploited a vulnerability previously unknown to the company to escape their sandbox and access the internet. The models correctly inferred that the answer to the evaluation was available on Hugging Face, a digital library of AI models and software, and broke into the company’s systems. Hugging Face detected the intrusion with its own AI models.

“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” OpenAI stated in a blog post about the hack.

Leave a Comment

Your email address will not be published. Required fields are marked *