We deleted our vector database

What it was for

We ran a vector database to support one feature: semantic similarity grading in evals. A grader takes the model answer and a reference answer, embeds both, and scores the distance. We stored the reference embeddings so we would not recompute them on every run.

It served about 400 queries a day. It cost $2,100 a month.

The measurement we did not do

Nobody chose that trade. The database arrived when semantic grading was a prototype, when we expected it to carry other features: retrieval over customer documents, trace search, a similarity view in the dashboard. None of those happened. What remained was one grader and a monthly invoice nobody looked at, because $2,100 is under every approval threshold we have.

When we finally measured, the numbers were embarrassing. The entire reference set was 240,000 vectors, which is small enough to hold in memory on a machine we already run. Queries were bursty and tolerant of latency, because they happen inside an eval run that takes minutes.

What replaced it

A flat file of vectors, loaded once per worker, searched with exact cosine similarity. Not approximate: exact, because at 240,000 vectors by 1,024 dimensions a brute force scan takes about 40 ms, and the approximate index we were paying for returned in 30 ms.

The file lives in object storage and is rebuilt when the references for a suite change, which is rare. The 40 ms sits inside an eval run, where nothing is waiting on it.

What we would do differently

Measure before choosing, obviously, but that is not the useful lesson, because everybody already agrees with it and does it anyway when the decision feels big. This decision did not feel big. It was a default: the feature involves embeddings, so it involves a vector database.

The useful version: write down the query rate and the corpus size you expect, then check them against reality on a schedule. Ours were off by two orders of magnitude and nothing in our process would have caught it, because no alert fires when a system is fine and unnecessary.

We now review the ten most expensive line items every quarter against what they actually serve. The first pass found this one and two others we are still arguing about.

The migration itself

Two days. Export the vectors, write them to a flat file, add a loader, replace the query call with a dot product over a matrix, and run both paths side by side for a week comparing scores. They agreed to eight decimal places on every pair, which they should, because exact search is exact.

The only real work was the rebuild trigger. Reference embeddings change when a suite changes, so the file needs to be regenerated and shipped to workers without a restart. That is a fifty line job and a cache invalidation, and we have learned to be nervous about those.

When you do need the database

This is not an argument that vector databases are useless. It is an argument that ours was serving a workload that did not need one, and that we never checked.

The thresholds where the answer flips are not mysterious. Millions of vectors rather than hundreds of thousands. Queries on a user facing path where 40 ms matters. A corpus that changes continuously rather than when someone edits a suite. Filtered search over metadata. Any one of those and the managed service earns its price immediately.

We hit none of them, and we would have known that in an afternoon at any point in the two years we were paying.

The quarterly review

The process change is duller than the migration and worth more. Once a quarter we list the ten most expensive infrastructure line items and write one sentence next to each: what it serves, how much of it, and what would replace it. Anything where the sentence is hard to write goes on a list.

The first pass found this database, an over provisioned search cluster serving the dashboard, and a staging environment that had been running at production size since 2024. Together they were about 9% of our infrastructure bill, all of it invisible because every individual line was small enough to ignore.