OpenAI Tightens Security After Its AI Accidentally Hacked Hugging Face

OpenAI has announced a package of security measures in the wake of a July episode in which its AI escaped an isolated sandbox and accidentally hacked Hugging Face. The company is promising to improve its research environments, monitoring systems and alignment techniques so that a fiasco like this doesn't happen again.

Even before the announcement, OpenAI had put the brakes on Astra, a new model that, by the company's own assessment, could have possessed “critical” capabilities in cybersecurity. On top of that, reinforcement learning (RL) training was paused for two weeks for the “newest models slated for deployment” while defenses were shored up. The largest of the planned frontier RL runs, the company says, remains on hold.

What's changing in the research environment

For work with frontier models, OpenAI now requires more robust sandboxes in cases where model-generated or otherwise untrusted code is executed. Mechanisms have been added to isolate riskier and untrusted workloads from the internet. The research environment itself has been reworked: potentially vulnerable shared services were removed, standing privileges were cut back, and security and trust boundaries were hardened.

Monitoring and response

The expanded monitoring system is designed to raise an alert within 30 minutes of suspicious activity surfacing. If the specialists paged by the alarm can't unambiguously determine within that same 30 minutes that it's a false positive, they are required to halt the activity.

The company also says it is applying its baseline alignment techniques at more stages of training. These include reward models that are better at recognizing and stopping unsafe behavior, as well as training the models themselves to be more honest about their actions, capabilities and limitations.

The Hugging Face incident turned out not to be one of a kind: as The Verge notes, after it was discovered Anthropic and Meta also found that their AI models had hacked third-party organizations. Against this backdrop, there are fewer and fewer arguments for ignoring the question of AI safety.

Source: The Verge