Why we route on p95, not p50

The number that looks good

Every latency dashboard starts with a median, because the median is the number that behaves. It is stable, it moves slowly, and it is almost always fine. Ours sits at 240 ms. If you shipped a product on the strength of that figure you would feel good about it right up until the support queue told you otherwise.

The median describes the request in the middle. It says nothing about the request at the end, and the request at the end is the one a person is waiting on. A four second call does not average out against a two hundred millisecond call. Nobody experiences an average. One user waited four seconds.

What we saw in the traces

We went looking for this after a customer told us their agent felt slow and their dashboard disagreed. Fernbank runs a support agent on top of us. Their median was 310 ms and had been for weeks. Their p95 was 4.1 seconds, and between one and four in the afternoon it was worse than that.

The shape of the distribution mattered more than the summary. Roughly 6% of their sessions crossed four seconds, and those sessions were abandoned at three times the rate of the rest. The median had no way of showing this. That is not a defect of the median. It is what a median is for.

Two things were producing the tail. The first was ordinary provider variance, which every provider has and none of them publish. The second was a degradation pattern in one provider that arrived in the afternoon and lasted about three hours.

Scoring on the tail

Our router picks a model per request. It scores every eligible model on cost, on quality from your own eval history, and on latency. The question is which latency number goes into that score.

We used to use a rolling median per model per region. It made the router look decisive: a model either was fast or it was not, and the number rarely moved. It also meant the router could not see a provider entering a bad afternoon until that afternoon had already moved the median, which took hours.

We now score on p95 over a fifteen minute window, and we keep p50 only for display. A provider whose tail widens loses position within a few minutes, long before its median moves at all. When the tail recovers, it climbs back.

The effect on our own numbers was smaller than the effect on the numbers that matter. Our p50 barely moved. Our p95 went from 1.9 seconds to 610 ms.

The arithmetic

Here is the part that changed our minds. Take a route that calls one model with a p95 of 4.1 seconds. If 5% of calls land in the tail and a session makes six calls, the chance that a session contains at least one tail call is 1 minus 0.95 to the sixth, which is 26%. A quarter of sessions contain a four second pause.

Now route the same traffic across two providers with a one second budget and failover. The probability that both are in a bad window at the same moment is not 5% of 5%, because provider incidents correlate more than we would like, but it is close enough to make the point. Measured over a month of Fernbank traffic, sessions containing a four second pause fell from 26% to under 2%.

None of that arithmetic is new. It is the same reasoning behind hedged requests in any distributed system. What is new is that it applies to model calls, where the variance is larger than anything you would tolerate from a database, and where most teams are still calling one provider directly.

Why a budget beats a timeout

A timeout is a decision you make once and apply everywhere. A budget is a decision the router makes per request, with the current state of every provider in front of it.

When you set a one second budget you are not asking us to cancel anything at one second. You are telling the scorer that a model whose p95 is over your budget is not eligible for this request unless nothing else is. That is a different instruction and it produces a different result: fewer retries, because fewer calls were sent somewhere slow in the first place.

Retries are the expensive way to fix a tail. You pay for the first call, you pay for the second, and the user waits for both. Fernbank added no retry logic at all. They set a budget, turned on failover, and the tail went away.

What we still get wrong

Scoring on p95 makes the router pessimistic about models with small sample sizes. A model we have called two hundred times has a noisy p95, and noise in the score reads as slowness. We damp it with a floor on sample size, which means a new model takes about a day to earn its position. We have not found a version of this that is both fast and honest.

We also do not score on p99 yet. The data is noisier still, and the failure it describes is better handled by a budget and a fallback than by scoring. If you disagree, we would like to hear why.

How we measure p95 without lying to ourselves

A percentile is only as good as its window and its population. Ours is computed per model, per region, over a fifteen minute sliding window. It excludes calls that failed before the provider accepted them, because a connection error is not a latency measurement. It includes calls that hit the budget, counted at the budget value, because excluding them would make a model look faster the worse it got. That last decision took an afternoon to argue and it is the one we would defend hardest.

We keep the window short on purpose. A one hour window is smoother and produces a number that looks better on a chart, and it is useless for routing, because by the time an hour of degradation has moved the number the incident is over and your users have already left.

The cost of a short window is noise on low volume models. We handle that with a sample floor: under two hundred calls in the window, a model inherits the p95 of its provider rather than its own. It is a blunt instrument and it is better than routing on four data points.

What this does to cost

Routing on the tail is not free. A model with a wide tail is often a cheap model, and scoring it down moves traffic to something more expensive. On our own traffic the effect was about 4% on average spend, which is real money at volume.

You decide where that trade sits. A policy that says cheapest above 0.90 with a one second budget will spend more than cheapest above 0.90 with no budget, and the difference is the price of the tail. We show both numbers in the dashboard so the trade is visible rather than implied.

Three numbers worth watching

Tail ratio, p95 divided by p50, per route. Under three is healthy. Over ten usually means something structural, a retry or a schema failure rather than a slow provider.

Tail share, the percentage of calls above your budget. This is the number to alert on, because it moves before the percentile does.

Session tail probability, the chance that a multi call session contains at least one call above budget. It is the only one of the three a user would recognise, and the one almost nobody computes.