Cloudflare: 155 AI responses leaked data from other customers' prompts

Summary

• 155 GLM-5.3 responses on Workers AI carried data from other customers' requests over 13 days.
• Cause: a prefill KV cache smaller than the model's advertised 1,048,576-token context window.
• Anyone running SGLang with disaggregated prefill/decode faces the same cross-request exposure risk.

The latest

A GLM-5.3 model on Cloudflare's Workers AI returned 155 responses containing data pulled from other customers' requests between August 27 and September 9, 2026, the company disclosed on October 26. Every affected response came from prompts longer than 681,280 tokens — the size of the prefill KV cache — against an advertised context window of 1,048,576 tokens. Cloudflare said it found no evidence of deliberate exploitation.

Details

• The root cause: Cloudflare said the prefill KV cache was provisioned smaller than the advertised context window, and an SGLang bug in disaggregated prefill/decode mode let a rejected request keep running instead of failing. Admission queues checked whether a request had finished rather than whether it had been rejected, so decode nodes read stale memory pages from earlier requests.
• How it surfaced: Gadi Evron of Knostic reported the flaw through Cloudflare's HackerOne program on September 8 at 23:11, after noticing that very long prompts sometimes returned text belonging to other users' sessions. Cloudflare reproduced the leak across two accounts within 25 minutes of declaring the incident.
• The 40 minutes: Cloudflare declared a security incident at 13:22 on September 9 and made the model private at 14:00, with requests returning HTTP 400. The root cause was identified at 14:35, all prefill nodes moved to the corrected configuration at 14:52, the deployment was rebuilt with an engine patch at 19:04, and the model returned publicly at 19:40.
• The timeline before: GLM-5.3 shipped on August 27 with the undersized cache and was announced in the Workers AI changelog on August 28 with a one-million-token context window. SGLang pull request #36988 was merged into main on September 6. Hardening across all production SGLang deployments ran from September 11 to 19.
• What Cloudflare fixed: The company matched GLM-5.3's configuration to GLM-5.2 and raised the prefill cache to about 2.04 million tokens, roughly double the context length. It also terminated rejected requests at the admission queue, added a decode-side check comparing transferred tokens against reserved ones, and stopped its load balancer cancelling requests mid-transfer on client disconnect.
• Its own blind spot: Cloudflare said its load balancer behavior and its long-context tests contributed to the failure, because those tests checked status codes and performance rather than output correctness. It called that a critical gap, and said page-level safety markers, formal verification of the RDMA KV transfer protocol, and a cache-size-versus-context-length check are planned.
• Upstream patches: The underlying issue #30233 was first reported on July 6, 2026. Fixes include #36988 terminating cancelled prefill requests before decode in v0.5.20, #38935 rejecting them at admission in v0.5.21, Cloudflare-proposed change #38704, deferred KV release on by default in #37454 and #41023, and a per-transfer KV checksum in #39500 awaiting release.
• What was not published: Cloudflare said a small number of customers received affected responses but gave no figure, and did not publish how those customers were notified or whether sensitive data was exposed. After excluding its own tests and the researcher's, it said the affected traffic looked like ordinary long-context usage, and no other Workers AI model was affected.
• Operator guidance: Cloudflare advised SGLang operators to match KV cache size to context length, upgrade to v0.5.22 or later, enable deferred KV release and bounded Mooncake packing, and turn on --kv-canary and transfer checksums where supported. Those detection options do not yet cover GLM-5.3, GLM-5.3 Flash or DeepSeek V4.
• The pricing: GLM-5.3 launched on Workers AI at $1.40 per million input tokens, $0.26 per million cached input tokens, and $4.40 per million output tokens.

Background

Disaggregated prefill/decode splits inference across separate nodes: one builds the key-value cache from the prompt, another generates output and reads that cache over RDMA. The design raises throughput on long prompts, but makes memory page ownership between the two stages a correctness boundary.

Between the lines

The exposure needed two independent failures to line up — an undersized cache and an engine that kept rejected requests alive — which is why only prompts above 681,280 tokens were hit. The sharper finding is the test gap Cloudflare named itself: long-context tests validating status codes and latency would pass cleanly while responses carried another customer's text. Two months passed between the upstream issue report in July and the production incident.

What's next

Watch the SGLang release carrying the merged per-transfer KV checksum (#39500), whether draft #46435 zeroing KV pages on release advances, and whether detection options are extended to GLM-5.3, GLM-5.3 Flash and DeepSeek V4.

Source: Cloudflare