OpenAI Shuts Down a Bid to Siphon Off Its Models' Reasoning

OpenAI said it has identified and disrupted a coordinated "distillation" campaign aimed at illicitly extracting the protected reasoning traces from its AI models. The core cluster of activity, dating back to the first week of July, is attributed to individuals associated with Moonshot AI, a Chinese AI company based in Beijing. OpenAI offered no technical evidence for the attribution, likely for security reasons.

The company stresses that this was not a breach: "The operators did not break our encryption, compromise a database, or gain direct access to stored user conversations." Instead, they manipulated model interactions so that protected reasoning could be reproduced in a form visible to the requester — at scale and in a coordinated way that violated the terms of service.

Timeline

The activity began on July 1, 2026, at low volume, then spiked on July 24 and 25, when 16,000 attempted requests using the relevant extraction pattern came from more than 4,000 users. Further investigation surfaced similar "prompt-pattern" activity across more than 15,000 users. The campaign was fully disrupted on July 28.

OpenAI describes the effort as adversarial distillation — the systematic, unauthorized use of one model's outputs to train, reproduce, or improve another. It has since deployed extra mitigations and banned the fraudulent accounts. It also closed a "pathway" that let anyone already holding another user's encrypted reasoning replay it and recover the contents, and added checks to hold back streamed output that might expose reasoning.

An architectural flaw

In a study published in August, researchers from MATS Research, the ELLIS Institute Tübingen, and Synk described an architectural vulnerability: encrypted reasoning traces turned out to be "fully compatible and interchangeable across different sessions, users, and models within a provider's ecosystem." By injecting a trace from a capable model into a weaker, less-safeguarded model from the same provider, an attacker can force it to decode and output the trace verbatim — without ever jailbreaking the stronger model directly.

The researchers note this enables large-scale private-data extraction, invisible prompt injection via payloads hidden inside encrypted blocks, and the leakage of hazardous information buried in the reasoning process, even when the model's visible answer rejects the harmful request.

Risks and context

OpenAI warns that adversarial distillation poses safety and even national-security risks: extracted reasoning could be used to train another model without preserving the safeguards applied to the original's user-facing outputs. At scale, it accelerates the transfer of advanced capabilities without the matching investment in safety — a worry that sharpens as models grow more capable in dual-use domains.

This is not the first distillation accusation against Moonshot AI. Last month, rival Anthropic alleged the company quietly relayed user requests to Claude instead of processing them with its own Kimi model, then showed Claude's answers back to users and retained a subset of the exchanges to train its own chain-of-thought model. That activity was tracked under the moniker GTG-16002.

Source: The Hacker News