Corvid Labs
Developer tools
Caught a 12% quality drop before release
Quality drop caught before release
Corvid ships prompt changes weekly. The eval gate found the regression their reviewers missed.
The problem
Prompt changes went out with code review and a vibe check. Twice, a quality regression reached users and was found by a support ticket rather than by a test.
What they did
They promoted 340 production traces into a golden set and wired plinth evals run into CI with a hard gate at 0.92.
What changed
Three weeks in, a one-word prompt edit dropped the score 12% and failed the build. Three more regressions have been caught since. None reached a user.
The eval gate has paid for the whole platform twice, and the second time it was my commit.
Ada Kowalczyk
Founding Engineer, Corvid Labs
By the numbers
340 golden cases
4 regressions caught in Q2
0 reached production
1M tokens a month, free forever.
No card, no trial clock, no sales call. Move to a paid plan when the invoice makes sense, or don’t.