The cache that made everything slower

The 70 milliseconds we wanted back

Model scoring used to run on the request path. Every call arrived, the router pulled the current latency and quality figures for every eligible model, scored them, and picked one. That work took about 70 ms, which was most of what we were adding to a call.

So we cached it. Scores for every model in every region, computed once, refreshed on a timer, read from memory on the request path. The change was about forty lines. In staging our overhead went from 70 ms to 4 ms and we felt clever for most of a Thursday.

What happened on Friday

p95 went up. Not by a little: the router’s own contribution to p95 went from 90 ms to 280 ms, and total p95 on affected routes moved by close to a second.

The refresh ran every 30 seconds. It recomputed every score in one pass, and while it was writing, readers took a lock. The lock was held for about 8 ms, which is nothing, except that it happened to every request in flight at the same moment, twice a minute, across every instance at once, because every instance had started its timer at deploy time.

We had built a small, regular, synchronised stall and shipped it as a performance improvement.

Three fixes, two of them wrong

The first fix was to shorten the lock. We moved to copy on write: build the new score table off to the side, swap a pointer. The stall went away. The numbers still looked wrong in a way we could not explain, and we spent a day on it before noticing the second problem.

The refresh was reading from the same metrics store the router used for its own health checks. Under load that store returned stale rows rather than fail, which is a reasonable default and a terrible thing to build a scorer on. For about four minutes after a provider degraded, the cache was serving scores computed from the state before the degradation. The router kept choosing a model that had already started failing.

That is the failure that matters. A cache that is briefly slow is an annoyance. A cache that is briefly wrong about which provider is healthy will route traffic into a fire.

The third fix was the one that stuck. Scores are now written by the metrics pipeline rather than pulled by the router, they carry the timestamp of the measurement they came from, and the router refuses any score older than 45 seconds and falls back to a live computation for the models involved. Refresh is every 15 seconds, staggered per instance by a random offset.

What it cost

Two days to ship, one weekend to understand, and one Monday to rewrite. Our overhead settled at 11 ms at p50, which is better than the 70 ms we started with and much worse than the 4 ms we briefly believed in.

The honest summary is that the first version optimised the number we were looking at, median overhead, and damaged the number we were not, tail latency and routing correctness under degradation. Cache changes are now reviewed against p95 and against a deliberate provider failure test before they go out.

The rule we wrote down

Any cache in the routing path has to answer three questions before it ships. How stale can this be before it is wrong rather than merely old. What happens to every reader at the moment it updates. And what does it serve when the thing it is caching is unavailable.

We got the third one right by accident the first time, which is why it took us until Sunday to find the second one.

How we test it now

Every change to the routing path runs against a provider failure harness before it ships. The harness holds traffic at a steady rate and then degrades one provider on a schedule: latency first, then error rate, then a hard failure, then recovery. We assert on three things. The router moves traffic within one refresh interval. No request waits on a lock for more than two milliseconds. And the scores the router used are never older than the staleness bound.

The harness found two regressions in the six months since, both in code that had nothing to do with caching and both of which would have shipped. A test that deliberately breaks a dependency is worth more than a test that checks the happy path faster.

On measuring the thing you are not changing

The deeper mistake was methodological. We had a benchmark for router overhead and no benchmark for router correctness under degradation, so we optimised the one we could see. The second number was not hard to measure. It had simply never been anyone’s job.

The fix was to put both numbers in the same place: overhead at p50, overhead at p95, and time to reroute after a provider degrades, on one dashboard, reviewed at the same moment. A change that improves one and damages another is now obvious in the review rather than on Friday afternoon.