fix(proxy): stream Server-Sent Events instead of buffering them for capture

[abezzub-dr]

What this fixes

Streamed responses time out and retry in a loop through the proxy, while the identical request works when sent directly. It surfaces on large or slow-to-start streaming requests — an LLM /v1/messages with a big context, or a cache-cold resume — where the upstream takes a while to produce its first bytes. The client shows repeated API error · Retrying, the turn never completes; un-proxied clients are unaffected.

Cause

The proxy sampled the response body for logging with a blocking read-ahead (io.ReadFull of up to MaxBodySize = 8 KB) before forwarding anything to the client. For an incrementally-produced response this withheld the status line and every chunk — including the keepalive pings the upstream sends during a long time-to-first-token — until 8 KB accumulated (or the whole stream ended, for sub-8 KB responses). The client received nothing, tripped its first-byte timeout, and retried. "Slow first byte" became "no bytes at all". This starved any slow text stream — Server-Sent Events, application/x-ndjson, chunked JSON — not just one media type.

Fix

Add capturingBody, which tees up to MaxBodySize into a buffer as the downstream consumer reads, so capture never blocks the forward path. Both response paths (TLS-intercept and plain-HTTP) use it for text responses — the same isTextContentType gate as before, now non-blocking. The full body streams through untouched; the logged sample is whatever flowed past. captureBody's blocking read is retained for request bodies, which are already fully available.

On the intercept path the response copy is owned by httputil.ReverseProxy, so the canonical log line for a text response is emitted from the body's Close (carrying the streamed sample and full duration). ReverseProxy closes the body unconditionally, so it fires exactly once; protocol upgrades (101) and non-text responses are logged synchronously instead. The plain-HTTP path captures during its own copy and logs synchronously after it.

Scope

This does not change the llm-gateway response-policy path: when a Keep engine is configured for a host, evaluateAndReplaceLLMResponse reads the full response (io.ReadAll, up to maxLLMResponseSize) to evaluate it before forwarding, so a stream on that host is still buffered end-to-end. Evaluating a full-response policy inherently needs the whole body; a streaming-aware llm.EvaluateStream path could lift this later.

Tests

• TestCapturingBody_NeverBlocksOnSlowStream — a body that emits one record then blocks must not trigger a read-ahead; uses a non-SSE application/x-ndjson stream.
• TestCapturingBody_StreamsAndCaptures — the full body passes through while a bounded sample is captured; onClose fires once.
• Intercept tests that assert on the canonical log line wait for the deferred entry.
• Full ./proxy suite passes under -race; go vet and gofmt clean. Confirmed end-to-end against a live deployment: looping sessions recovered, and a ~1 MB-context turn completed in single-digit seconds.