Encryption Slips Past Grok's Defenses: The Chatbot Leaks User Data

Prompt injection attacks remain one of the thorniest problems facing large language models: a chatbot can't reliably tell a user's instructions from stray text it picks up while, say, reviewing an email or a web page. Developers are forced to bolt protective filters onto the model — as the article aptly puts it, this is like putting up a guardrail on a dangerous curve instead of engineering the road properly in the first place.

Rony Utevsky of Adversa found a disarmingly simple way to vault over that guardrail. The malicious instruction isn't written out in the clear — it's encrypted. On the same page, right next to the ciphertext, sit an open decryption instruction and the key. The moment a user asks Grok to summarize the page's contents, the model dutifully carries out the command — no warning, no request for confirmation.

The key that isn't there

The decrypted instruction tells the model to “assemble the decryption key.” In reality, what gets substituted in for this supposed key is the user's name, their location and their chat history. That value is then tacked on as a parameter to a link pointing to the attacker's server. Grok opens the link — and the private data ends up in the attacker's logs.

Why does the model refuse to carry out the same instruction in plain text yet happily execute it in encrypted form? Adversa won't say for certain, but they offer a plausible hypothesis. Grok's filter checks the text going into and coming out of the model, but it doesn't peer into the results of the model's own code. An instruction to process ciphertext with the PBKDF2 and AES-256-GCM algorithms passes as a harmless request: the classifier reads it but doesn't execute it, and so has no idea what lies behind it. And once the instruction is decrypted inside the sandbox, it comes back to the model as its own output — and the model acts on it, bypassing any check.

“Static defense mechanisms classify input data as text, but don't execute it,” Utevsky explains. Everything the filter needs is sitting right there on the page, but to reconstruct the original text you have to run the cryptography — something no content classifier does during a check.

Not just Grok

Adversa tried a similar trick on Gemini: there the decrypted text was disguised as an error message, and the embedded instruction told the model to “read the error and act on it.” The result was that the model produced forbidden content, and with a modified payload it even gave up its own system instructions, which it is not allowed to reveal. Google made no comment on Gemini's behavior: jailbreaks aren't covered by their bug bounty program. That said, over the past few weeks Gemini has become noticeably more resistant to the attack, though whether the cause was a filter update or a change of model version couldn't be established.

The researchers dubbed their method cryptographic context injection and see it as a specific instance of a broader shift: attacks that manipulate not the request itself, but the entire context the model treats as “its own” — tool output, intermediate results, execution state. This attack surface is far wider than the familiar “model input,” and that, Adversa believes, is where the next generation of threats will be born. For now the cycle repeats: defenders patch one hole, attackers find another.

Source: Ars Technica