From the team
Notes from people who are on call.
Engineering write-ups, mostly about things that went wrong. No thought leadership, no roundups, no predictions for next year.
Why we route on p95, not p50
Median latency is the number that makes your dashboard look good and your users leave. Here is the arithmetic that changed how our router scores models.
Priya Nandakumar
8 min read
The cache that made everything slower
We cached model scores to save 70 ms. It cost us 190 ms and a weekend. A short story about cache invalidation being one of the two hard problems.
Sam Oyelaran
6 min read
What 40 million evals taught us about temperature
We ran the same suite at eleven temperature settings across seven models. The results were less interesting than the variance, which was the point.
Ada Fitzgerald
11 min read
Zero-day retention, and what it cost us to build
Not storing something is harder than storing it. Nine months, one rewritten trace pipeline, and every debugging tool we had to give up.
Marcus Reinholt
9 min read
Streaming is a distributed systems problem
A failover halfway through a stream is not a retry, it is a partial-order problem. What we got wrong the first three times.
Sam Oyelaran
7 min read
We deleted our vector database
It served 400 queries a day and cost $2,100 a month. What we replaced it with, and why we should have measured first.
Ada Fitzgerald
5 min read