Streaming is a distributed systems problem

A stream is a promise you keep making

A blocking call has one moment of failure. You either get the answer or you do not, and if you do not, you try again. Nobody has seen anything yet, so a retry costs latency and nothing else.

A stream has as many moments of failure as it has tokens. By the time a provider dies, the user has read three sentences. A retry that starts from the beginning produces those three sentences a second time, worded differently, which is worse than an error.

What we got wrong the first time

Version one treated failover as a retry. If the provider errored, we called the next model with the original request and streamed the new answer. It worked in tests, because in tests the failure happened before the first token. In production it produced duplicated text often enough that customers turned streaming off.

Version two: replay the prefix

The second attempt held what we had already sent and passed it to the second model as an assistant prefix, asking it to continue. This is the obvious fix and it half works. Models are inconsistent about continuing someone else’s sentence, and the join is visible: a change of register mid paragraph, or a repeated clause.

It also introduced a correctness problem that took us a while to see. If the first model had already emitted a partial JSON object that the schema validator was going to reject, the second model inherited a malformed prefix and produced something that validated against nothing.

Version three: commit at boundaries

What we run now buffers at semantic boundaries rather than at tokens. For plain text the boundary is a sentence. For structured output the boundary is a complete value in the schema. Tokens stream to the client as they arrive, but the failover record only advances at a boundary, and a failover resumes from the last committed boundary rather than the last byte.

If the failure happens mid sentence, the client has seen a partial sentence that will be emitted again. We chose to accept a repeated fragment of at most one sentence rather than a repeated paragraph or an incoherent join.

For structured output we do not stream partial values to the client at all. Fields appear as they complete. That costs perceived speed on a large object and it removes an entire class of failure where a consumer parses half a number.

The bug we shipped anyway

In July we fixed a case where a provider failed after the stream opened but before the first token. The retry started from the second token, because the boundary logic recorded the open as a commit. It affected roughly one streamed call in forty thousand. We found it because a customer eval suite caught a class of truncated answers, which is a good argument for running evals on streamed output and not only on blocking calls.

It is fixed, the traces were backfilled, and there is a test for it now.

What to take from this

If you are building failover over streams, decide your commit boundary before you write any of it, and decide what a client is allowed to see before it is committed. Those two decisions are the whole design. Everything else is bookkeeping.

And run your evals against the streamed path separately. The blocking path and the streaming path are different programs, and ours disagreed with each other for about three weeks.

What the client sees during a failover

Nothing, if we did it right. The connection stays open, the token stream pauses for the time it takes to open a second call, and then continues. Median pause on a failover is 340 ms, which is noticeable in a chat interface and much less noticeable than an error.

We deliberately do not send an event saying a failover happened, because every client we tested rendered it as an error. It is in the trace instead, which is where it belongs: your logs know, your user does not.

If you want the client to know, the metadata is available on the final chunk. Two customers use it, both to annotate their own logs, neither to change what the user sees.

Why boundaries are cheaper than they look

Holding a sentence in memory sounds expensive until you compare it to the alternative, which is holding the whole response. A sentence is a few hundred bytes. The whole response can be tens of kilobytes, times every concurrent stream, and under zero day retention it is also a compliance question rather than a memory question.

Boundary commits also gave us something we did not plan for: a natural place to run streaming evals. A grader that needs a complete sentence can run at each boundary instead of waiting for the whole response, so a structured output that is going to fail its schema can be caught partway through and re routed rather than returned and rejected.

The part that is still unsolved

Tool calls. A stream that includes a tool call has a boundary that is not textual, and the tool may already have run by the time the provider fails. Re running the call on a second model means either re running the tool, which may not be idempotent, or splicing a result the second model did not produce, which is its own category of strange.

Today we do not fail over after a tool call has executed. The stream errors and the client decides. That is the honest behaviour and it is not a good answer, and it is the next thing on this list.