How to Steal the Hidden “Reasoning” of Proprietary Language Models

The leading providers of large language models have long hidden their systems' step-by-step “reasoning” — the so-called chain-of-thought. This is done to protect intellectual property and to limit information leakage. Yet the reasoning itself doesn't actually vanish: instead of storing it on the server, providers return it to the client as blocks of encrypted text, which the client then attaches to every subsequent request.

A new study titled “Stealing Reasoning Traces from Proprietary LLM APIs” points to an architectural flaw in this scheme. It turns out that the encrypted blocks are fully compatible and interchangeable across different sessions, users, and even different models within a single provider's ecosystem.

Decryption by proxy

On this compatibility the authors built a scalable “jailbreak decryptor.” The idea is simple and, for that very reason, unsettling: an encrypted reasoning trace from a strong model is slipped to a weaker, less well-guarded model from the same provider — and that model obediently decrypts and outputs the text verbatim. There's no need to attack the more capable model directly at all.

The researchers identify four attack vectors. The first is bypassing anti-“distillation” safeguards: someone else's proprietary reasoning can be extracted, as demonstrated on models from Anthropic, OpenAI, and Google. The second is the mass leakage of private data. Developers often post their session logs publicly, unaware of what lies hidden inside the encrypted blocks. By decrypting 315,320 such blocks collected from public repositories, the authors recovered 367 fragments of personal data and 182 sets of credentials.

The third vector is especially intriguing: the method brings to light dangerous information tucked away in the reasoning process, even when the model's final visible answer politely declines the malicious request. In other words, the model “thought” about the forbidden thing, refused to say it out loud — but the thought remained in the encrypted block. The fourth is “invisible” command injection: a malicious payload can be hidden entirely inside an encrypted block, thereby poisoning public agentic workflows.

What comes next

The authors of the work went through a responsible disclosure process and proposed concrete protective measures — both cryptographic and at the level of the system architecture — to secure reasoning that is processed on the client side. For now, though, the takeaway is rather sobering: the attempt to hide the model's “thinking” by offloading the storage of the encrypted context onto the client has spawned a new class of vulnerabilities instead of reliable protection.

Source: Schneier on Security