AI Guardrails Are Getting in the Way of Legitimate Security Researchers

The major AI companies have built a system of checks and guardrails around their models to lock out bad actors. But as TechCrunch found after talking to several security researchers, those same barriers are hurting legitimate specialists — both network defenders and people working in offensive security.

The conversation was prompted by an episode in June: the US government imposed export restrictions on Anthropic's Mythos and Fable models — partly because of a report on the possibility of bypassing their safeguards and using them for cyberattacks. The restrictions were later lifted: Fable 5 returned to general availability on July 1, while Mythos 5 is so far available only to vetted American organizations. Anthropic, for its part, has for years positioned Mythos itself as a “cyber doomsday machine” that can be entrusted only to a carefully selected set of users.

A tool that can't be split apart

Both Anthropic and OpenAI offer researchers verification programs (Trusted Access for Cyber and the Cyber Verification Program) that grant access to models with relaxed restrictions. But many take issue with the approach itself. The well-known researcher Mark Dowd admits he's uncomfortable with the fact that “random large companies get to make arbitrary decisions about what's acceptable in security and what isn't.”

Chris Anley, chief scientist at NCC Group, explains the crux of the problem: the request “fix this code” is at once a defensive mechanism and a map for finding critical vulnerabilities. One and the same tool inseparably serves both defense and offense — “like a hammer: you can't build a house without it, but it's inevitably a weapon too.” When a model refuses to answer, it's the defenders who suffer first.

Where the researchers are going

Some specialists have developed workarounds. At CrowdFense, frontier models are used only for reverse engineering, while vulnerability hunting is done on local open-source models — so that confidential data doesn't leak into the cloud and end up in future training sets. Researcher Giuseppe Cali doesn't trust AI with the bug hunt itself at all: “I'm jealous of my bugs, and I love this game too much to hand it over to the models.”

An anonymous source at a maker of smartphone components said that without taking part in Anthropic's program the tools are nearly useless: “The moment the model catches a whiff that we're doing something security-related, it just stops.”

Chris Thompson, head of RemoteThreat, notes that the guardrails work inconsistently and “differently every day,” so that instead of analyzing vulnerabilities researchers spend their time “negotiating with the model.” The upshot, he says, is paradoxical: responsible specialists are being pushed out of US-controlled systems and toward foreign ones — for example, freely downloadable Chinese models like GLM, which have no checks and no restrictions at all.

Thompson urges the labs not to tighten the screws but to open up their programs, grant responsible access, and punish those who abuse it. Otherwise, he warns, defenders will meet the coming wave of attacks “of unprecedented scale and speed” with their hands tied.

Source: TechCrunch