Crossing an encrypted reasoning trace into a weaker model

FACT (primary report, retrieved and read in full). On 2026-09-30 OpenAI published "Disrupting a coordinated model-distillation campaign", stating that it had identified and disrupted a coordinated campaign designed to extract protected reasoning from its models.

FACT — OpenAI's own definition of the target. The report defines the asset under attack:

"Protected reasoning is the model's internal record for working through a task; extracting it can reveal information withheld from the final answer and help others reproduce the model's capabilities."

FACT — this was not a breach. OpenAI was explicit and it matters:

"The operators did not break our encryption, compromise a database, or gain direct access to stored user conversations. Instead, they manipulated model interactions so that protected reasoning could be reproduced in forms visible to the requester in a coordinated, scaled manner that violated our terms of service."

FACT — the technique. Operators used a novel method:

"We saw operators attempt to extract protected reasoning in novel ways, including by copying encrypted reasoning from one conversation and asking a model in another conversation to decrypt and transcribe the hidden reasoning content."

COMPANY CLAIM — the campaign timeline, exactly as OpenAI states it.

"The activity began on July 1, initially at a low volume until we observed high-volume spikes on July 24 and 25 consisting of 16,000 requests using a relevant extraction pattern from over 4,000 users. Further investigation identified related prompt-pattern activity across a cluster of more than 15,000 users, which we fully disrupted by July 28."

Every one of these five figures (1 July · 24–25 July · 16,000 · >4,000 · >15,000 · 28 July) was checked against the primary text and independently reproduced by CNBC, CyberScoop and The Hacker News. They are confirmed as accurate reports of what OpenAI said. They are not confirmed as accurate accounts of what happened.

COMPANY CLAIM — the attribution, quoted in full and without paraphrase:

"It is unclear whether all operators we observed during the relevant time period originated from a single actor. However, we attribute a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi."

FACT — OpenAI published no evidence for the attribution. This is the single most important finding in this research, and it was independently established by two outlets on the day of publication:

• CyberScoop (Derek B. Johnson, 2026-09-30): "OpenAI's blog post does not cite any technical evidence or reasoning for its attribution." OpenAI told CyberScoop it was not sharing further information "for security reasons."
• The Hacker News (Ravie Lakshmanan, 2026-10-01): "It did not cite any technical evidence to back this assessment, likely owing to security reasons."

FACT — Moonshot did not respond. CNBC: "Moonshot did not immediately respond to CNBC's requests for comment." CyberScoop: "CyberScoop has reached out to Moonshot AI for further comment." No response from Moonshot AI, and no denial, appears in any source retrieved for this story.

FACT — the vulnerability class is independently real. OpenAI confirmed the attack paths were genuine and credited outside researchers:

"Independent security researchers also brought related cross-model and conversation-compaction vulnerabilities to our attention through responsible disclosure. We investigated their findings and confirmed that the attack paths they identified were real."

FACT — the change in the attack, not just in the news. Before this campaign, adversaries trying to clone a frontier model were largely confined to harvesting visible outputs — answers, which are by design the public face of the model. This campaign targeted protected reasoning: the model's internal record of how it worked through a task, which providers deliberately withhold.

FACT — the change is architectural, not behavioural. The researchers OpenAI credited, "Stealing Reasoning Traces from Proprietary LLM APIs" (arXiv:2608.09867, submitted 2026-08-10), identify the reason it worked. Leading providers conceal step-by-step reasoning and return the traces to the client as blocks of encrypted text, which the client passes back with each subsequent request:

"Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with each subsequent request. Building on prior research, we identify an architectural vulnerability: these encrypted blocks are fully compatible and interchangeable across different sessions, users, and models within a provider's ecosystem."

INDEPENDENTLY VERIFIED — the exploit, in the researchers' own words:

"By injecting an encrypted reasoning trace from a given model into a weaker, and less safeguarded model from the same provider, we force it to decode and output the trace verbatim in plaintext, without ever jailbreaking the more capable model directly."

INTERPRETATION. This is the real story of the story. The encryption on reasoning traces was doing security work, not confidentiality work: it was assumed to be safe because the capable model cannot be talked out of revealing it. The weak-model oracle removes that assumption entirely. The strong model's safeguard is bypassed without ever attacking it, so every guardrail tuned against the frontier model — refusals, classifiers, rate limits on the flagship endpoint — is aimed at the wrong target. That is a category error, not a tuning failure.

FACT — the attack had collateral consequences beyond distillation. The same research demonstrates four distinct vectors, only the first of which is distillation:

1. Circumventing anti-distillation mechanisms (demonstrated across Anthropic, OpenAI and Google).
2. Large-scale private data extraction — "By decoding 315,320 reasoning blocks scraped from public repositories, we recovered 367 Personally Identifiable Information (PII) artifacts and 182 credentials." Developers who share session logs publicly, unaware of what the encrypted blocks contain, are leaking secrets into their own repositories.
3. Hazardous information disclosure, "even in cases where the model's final, visible output safely rejects a malicious request" — safety assessed on the answer misses what happened inside the reasoning.
4. Invisible prompt injection — "embedding malicious payloads entirely within encrypted blocks to poison public agentic rollouts."

INTERPRETATION. Vector 2 is the one most likely to cause concrete downstream harm, and it has nothing to do with model theft. Anyone who has committed a reasoning-bearing session trace to a public repository in the past year should assume it may be decryptable.

COMPANY CLAIM — what OpenAI says it did about it. Banned or restricted fraudulent accounts; strengthened signup and infrastructure controls; expanded monitoring of related networks; "closed a pathway that allowed someone who already possessed another user's encrypted reasoning to replay it and recover its contents"; "added checks to detect and hold streamed output that might expose reasoning"; worked with third-party providers whose services carried related activity; and shared findings through the Frontier Model Forum and government information-sharing channels.

FACT — scope was broadened deliberately. OpenAI states the manipulation "is not a vulnerability unique to OpenAI's models" and flags a general exposure: "Systems that support portable or replayable reasoning artifacts may face related risks." OpenAI also committed that "Partner-hosted deployments need the same protections as first-party services."

INDEPENDENTLY VERIFIED (arXiv:2608.09867). The mechanism, step by step:

1. Reasoning leaves the provider by design. To keep traces out of server-side storage, providers ship them to the client as encrypted blocks, and the client returns them on each subsequent request. The client therefore holds the protected reasoning as an opaque token.
2. The blocks are not scoped. Encryption binds the trace to the provider, not to the session, the user, or the model. The research finding is that these blocks are "fully compatible and interchangeable across different sessions, users, and models within a provider's ecosystem."
3. Cross-model replay. An encrypted trace produced by a frontier model is submitted into a conversation with a weaker, less safeguarded model from the same provider.
4. The weak model decodes it. The weaker model transcribes the foreign trace verbatim in plaintext. The research is explicit that this works "without ever jailbreaking the more capable model directly."
5. Scale. Because the primitive is "paste a block," it is trivially parallelisable — which is how a ~16,000-request campaign is coherent as an operation at all.
6. No key is stolen. Confidentiality was never the attack. This is an authorisation and scope-binding failure in the reasoning-artifact design, not a cryptographic break.

FACT — OpenAI's confirmation is narrower than the researchers'. OpenAI says it closed "a pathway that allowed someone who already possessed another user's encrypted reasoning to replay it and recover its contents." Note the phrasing: OpenAI describes closing replay of another user's already-possessed trace. It does not claim to have bound encrypted reasoning blocks to session, user and model. The cross-model weakness — the researchers' core claim — is not what OpenAI's remediation sentence claims to fix.

INTERPRETATION. That gap between the researchers' architectural finding and OpenAI's narrow "pathway closed" claim is the most consequential unresolved technical question in this story (§13, §14). If blocks remain interchangeable across models within a provider, the weak-model oracle is still open and the mitigation is perimeter control, not a fix.

CONTEXT — the same class, older. Anthropic, 2026-02-23: "In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude's reasoning traces." Google Cloud GTIG, 2026-02-13, on coercing Gemini's thinking content. OpenAI's disclosure is the first to name a specific technique rather than a target.

INTERPRETATION — the moat question. Distilling a frontier model's answers copies its performance. Distilling its reasoning traces copies its method — including the scaffolding that makes reasoning models work. OpenAI states the stakes: "Extracted reasoning could be used to train another model without preserving the safeguards applied to the original model's user-facing outputs," and "at scale, distillation can also accelerate the transfer of advanced capabilities without requiring the same investment in safety."

INTERPRETATION — the escalation OpenAI itself names. Guardrails are enforced on user-facing outputs. Extracted reasoning is the input to training the next model, and that model never inherits the safeguards, because safeguards are applied as output filtering rather than embedded as trainable behaviour. This is the argument for safety-in-weight, not safety-in-filter.

FACT — the geopolitical layer is real but not new. CNSA/CISA/NSA-level documents already framed this as national-security infrastructure theft. CISA: these campaigns "form the core—not merely a supplement—of their AI development strategy", and companies that do this "see significantly shorter AI development timelines and reduced financial expenditures in training a frontier model." What is new is a US frontier lab publicly blaming a named Chinese competitor by company name for a reasoning-extraction campaign.

INTERPRETATION — what is genuinely novel, and it is not the accusation. Three labs have now accused Moonshot of distillation. Naming a competitor is the least new part. The new part is the technical disclosure of an attack that defeats reasoning concealment without touching the protected model — and that disclosure is being handed, in the same document, to the entire industry including the attackers.

INTERPRETATION — the disclosure dilemma. OpenAI states it published after deploying mitigations and consulting researchers and partners. But it also published a working method. Every other frontier lab now has the recipe, and so does anyone who reads this story. OpenAI's own framing concedes the trade: "We believe sharing what we have learned now will help the broader ecosystem strengthen its defenses." That is a genuine argument. It is also an argument made by the party that benefits most from the resulting normalisation of "publish your breaches."

COMPANY CLAIM, now INDEPENDENTLY VERIFIED as capability. Before this disclosure, extracting reasoning meant prompting a capable model until it narrated its own thinking, with refusals to fight. That barrier is now documented as bypassable by design. What became possible:

• Reasoning-trace extraction at scale without attacking the frontier model. No jailbreak of the flagship is required.
• Reasoning-trace harvesting from providers who do not narrate reasoning, by borrowing a decryption oracle from the same provider's weaker model.
• Passive secret leakage for anyone who has committed reasoning-bearing session traces to public repositories. Quantified by the researchers: 315,320 blocks → 367 PII artifacts and 182 credentials.
• Safety assessment by output inspection is insufficient, where the hazardous content exists in the reasoning that the output filter never saw.
• Invisible prompt injection via payloads carried inside encrypted blocks, targeting shared agentic rollouts.
• Cross-provider applicability. Demonstrated across Anthropic, OpenAI and Google.

INTERPRETATION — for defenders, the practical unlock is the detection surface, not the defence. CISA's advisory converts this into enterprise action, and its second recommendation is the one that will be controversial:

"Deploy targeted response changes: Subtly alter responses for suspected malicious distillation attempts to attenuate the payoffs to companies conducting industrial-scale distillation campaigns."

Including: "Reducing reasoning depth, presenting correct information with different reasoning, or stylistic inconsistencies may evade detection while reducing training usefulness" — and critically, "Avoid informing China-based AI company users suspected of distillation campaigns of a switch to a downgraded model." A US government agency has officially recommended silently degrading responses to suspected adversaries. That is a significant normalisation, and it sits uncomfortably beside model-transparency commitments.

1. Will Moonshot AI respond, deny, or sue? An unanswered public accusation of industrial-scale theft against a named company is unusual and unlikely to end in silence indefinitely.
2. Did OpenAI bind encrypted reasoning blocks to session, user and model cryptographically — or only close the replay pathway? This is the question that determines whether the researchers' attack still works. Unanswered in every source retrieved. (§5, §8)
3. What is the extraction yield? Attempted-versus-successful is acknowledged; the successful fraction is not disclosed.
4. Why 16,000 requests? Either this was a capability probe, an early-stage operation disrupted before scaling, or a small operation. OpenAI gives no read.
5. Was the July campaign in fact using the technique the researchers were simultaneously disclosing? The campaign began 1 July; the paper was submitted 10 August; OpenAI credits responsible disclosure without dates. Priority is genuinely unclear and is not established by anything retrieved.
6. Why did OpenAI not meet Anthropic's disclosure standard? Anthropic published IP correlation, request metadata, infrastructure indicators and named-profile matching. OpenAI published nothing and cited security reasons.
7. Has any frontier lab bound reasoning blocks to model identity? If not, the weak-model oracle is industry-wide and live.
8. What happened with the 182 credentials and 367 PII artifacts? Disclosed, notified, rotated, quietly patched?
9. Will providers actually implement CISA's silent degradation? And on what evidentiary standard, given that OpenAI's own attribution this week met none?
10. Is there a regulatory or legal path from a supported attribution to actual consequence for a named competitor?
11. Did the Frontier Model Forum sharing produce anything verifiable, or is coordination still narrative?

FACT — what OpenAI has already committed to. Three stated areas of continuing work: "stronger technical protections against extraction, better detection and enforcement against coordinated campaigns, and deeper threat-information sharing across industry and government." Plus named open items: partner-hosted deployment parity, and tool-output attacks requiring protections that "examine more than ordinary visible text."

PREDICTION — near-term (weeks to months), with reasons:

1. Moonshot AI eventually issues a response, and it will be a denial. A named accusation of industrial-scale theft without rebuttal is untenable for a company of its profile and funding access. (Rationale: reputational cost of silence grows; denial is the only available move absent evidence.)
2. At least one other frontier lab publishes a response to the arXiv finding within weeks, either acknowledging the weakness or disputing the characterisation. The paper demonstrated across three named providers; all three have reputational reason to engage. (Rationale: demonstrated-and-public findings create obligation.)
3. No provider confirms it has cryptographically bound reasoning blocks to session, user and model. (Rationale: this is a breaking client-protocol change requiring coordinated rollout across every SDK and agent framework. OpenAI's own remediation language describes closing a pathway, not re-architecting the artifact.) This is the most consequential prediction in this section — it is the reason item 2 in §16 matters.
4. A second, larger campaign is discovered and reported, most likely by a lab that coordinated more tightly with law enforcement first. The 16,000-request scale suggests this one was either an early probe or was cut off before scaling. (Rationale: OpenAI states work is unfinished and that it expects more sophisticated attempts; perimeter-only controls do not remove the primitive.)
5. At least one provider quietly changes reasoning output behaviour without announcing it, per CISA's guidance to vary responses and avoid informing suspected distillers. (Rationale: CISA recommended it explicitly and a US frontier lab has public reason to comply.)
6. A regulator or law-enforcement body acts on CISA's advisory — sanctions, export-control tightening, or a formal information request to a named company. (Rationale: the advisory names six firms with legal entities and recommends coordinated ecosystem response; it is co-sealed by NSA and FBI, which is the strongest institutional signal available.)

INTERPRETATION — the thing most likely to be missed. The industry response will focus on attribution politics — who accused whom — because that is the legible, newsworthy part. The technically decisive thread is narrower and quieter: whether encrypted reasoning blocks are still interchangeable across models within a provider. If they are, the weak-model oracle works, the published mitigations are perimeter-only, and every frontier lab has a known open extraction path. Nothing in this story's evidence resolves that question, and it is the one worth tracking.

OpenAI disclosed a real and important attack, in a document that also contains an unsupported accusation, about a campaign that was small.

Three things should be kept apart, because the story invites collapsing them.

One — the technical finding is the story, and it is strong. Researchers found that encrypted reasoning blocks are interchangeable across sessions, users and models within a provider, and used a weaker model as a decryption oracle to bypass anti-distillation protections without ever attacking the capable model. OpenAI independently confirmed the attack paths were real. That is a genuine architectural defect with independently demonstrated consequences: 315,320 blocks scraped from public repositories yielded 367 PII artifacts and 182 credentials. Concrete, verifiable, actionable — and independent of every disputed claim below.

Two — the attribution is not. OpenAI named Moonshot AI as behind a "core cluster" of the activity and published no technical evidence or reasoning, declining to elaborate "for security reasons." CyberScoop and The Hacker News both noted the absence of evidence on the day of publication. Moonshot AI did not respond to CNBC or CyberScoop. This should be reported as a company claim — a serious and consequential one, made on the record, but a claim.

Three — the CISA advisory is not about this. AA26-251A is dated 2026-09-08, twenty-two days before OpenAI's disclosure. It cannot corroborate this campaign or this attribution. It does independently corroborate the broader pattern: Moonshot AI named, alongside five other Chinese firms, in industrial-scale distillation of US models since at least mid-2025 — a claim Anthropic independently supports, and does so with published telemetry. Combining "CISA named Moonshot" with "OpenAI blamed Moonshot for a specific July attack" produces a story that feels far better sourced than it is.

On the numbers. 16,000 requests over two days is real and correctly reported, and it is small — roughly 0.5% of the 3.4 million exchanges Anthropic attributed to Moonshot, and orders of magnitude below CISA's "billions of tokens across millions of exchanges." OpenAI's own footnote says these are attempted, not necessarily successful, extractions. No yield is disclosed. The significance here is the technique, not the throughput, and coverage that treats 16,000 as a devastating number has been misled by the headline.

The deeper editorial point. The most consequential sentence in OpenAI's report is not the accusation. It is: "We closed a pathway that allowed someone who already possessed another user's encrypted reasoning to replay it and recover its contents." That describes closing replay. The researchers' finding is about cross-model interchangeability. The mitigation sentence and the vulnerability finding are not about the same thing, and no source retrieved says whether the cross-model weakness was fixed. That gap is the story to watch — not the geopolitics, which is loud and largely a repetition of claims made three weeks and seven months ago.

The angle worth taking. On the same day a US government agency recommended silently degrading model responses to suspected adversaries without telling them, one of the largest US AI labs publicly accused a named foreign competitor of industrial-scale theft and refused to say on what evidence. Both are signs that attribution is outrunning evidence in AI security — in enforcement and in accusation, at once. Watch that ratio. It is the honest measure of whether any of this is actually being secured.