There is a moment in almost every AI project where someone runs a prototype in front of the room, the output looks impressive, and the feature quietly moves onto the release schedule. Our team has learned to distrust that moment. A good demo proves the system can produce a good answer. It tells you nothing about how often it does, or what it does on the inputs nobody thought to try.
So we changed the order of work. Before we build the feature, we write down how we will judge it — not as a vague quality bar in a kickoff document, but as an actual file of inputs and expected behaviour, versioned in the repository next to the code. We call it the eval set. On most of our projects it has ended up being the closest thing we have to a real specification.
Why a demo is not evidence
In conventional software, one passing test is meaningful because the system is deterministic. If the invoice parser extracts the VAT number correctly once, it will do it again on the same input. Language models break that assumption. The same prompt can produce a different answer tomorrow, and a slightly reworded input can produce a completely different one today. The useful question is not "does it work." It is "on what share of realistic inputs does it work, and when it fails, how bad is the failure."
That reframing changes who needs to be in the room. Accuracy targets stop being an engineering detail and become a business decision. If a document-classification step is right 92% of the time, someone has to decide whether the remaining 8% is absorbed by a human reviewer, routed to a fallback queue, or whether the whole idea is not worth shipping. We would rather have that conversation before six weeks of build than after.
Where the first fifty examples come from
The single most common mistake we see is teams generating synthetic test cases with a model and calling it an eval set. It feels productive and it is nearly worthless, because the generated inputs share the model's own assumptions about what the data looks like.
Real inputs come from the process that already exists. If a company wants to automate part of its support inbox, the eval set is built from actual tickets. If it is supplier invoices, we ask for a folder of real ones — including the scanned, rotated, handwritten-in-the-margin specimens that the finance team complains about. Fifty examples pulled from production reality will teach you more than five hundred invented ones.
We deliberately oversample the ugly cases. A set that mirrors production distribution is mostly easy cases, and a model that handles the easy 80% will look excellent while quietly failing on the slice that actually costs money. So we weight toward the edges: ambiguous requests, mixed-language text, multi-page documents where the relevant number appears twice with different values, the customer who asks three things in one message.
Three ways to grade, in order of preference
Not every check needs a model behind it, and we reach for the cheapest one that answers the question.
- Deterministic checks where the answer is a fact. Did it extract the correct invoice total? Is the returned date in ISO format? Does the JSON validate against the schema? These are ordinary assertions, they run in milliseconds, and they catch a surprising share of real regressions.
- Rubric grading by a second model where the answer is open-ended. A support reply cannot be string-matched, but a grader can be asked specific, narrow questions: does the reply address the customer's actual question, does it invent a policy that was not in the provided documents, does it commit the company to a refund. Narrow questions produce far more stable grades than asking a model to rate quality from one to ten.
- Human review for the slice that neither of the above covers, and as a periodic audit of whether the model-graders still agree with people. We keep this sample small and regular rather than large and occasional.
Track a failure taxonomy, not an accuracy number
A single accuracy percentage hides the thing you need to know. Two versions of a feature can both score 91% while failing in entirely different ways, and one of those ways might be unacceptable.
We categorise failures instead. In extraction work the buckets are usually something like: wrong value extracted, value missed entirely, hallucinated value that does not appear in the source, correct value in the wrong field. Those four have very different consequences. A missed value stops the pipeline and a human notices. A confidently hallucinated value passes straight through into an accounting system. We will happily accept a version that misses more and invents less.
The number that matters is rarely overall accuracy. It is the rate of the one failure mode that nobody downstream will catch.
What this buys on the commercial side
An eval set turns most AI decisions from argument into measurement. When a new model version appears, we run the set and see the delta in an afternoon instead of debating it. When someone proposes a cheaper model for a high-volume step, the question of whether the quality drop is tolerable has an answer rather than an opinion. Prompt changes stop being a matter of taste, which matters more than it sounds — prompt edits are the easiest thing in the stack to change and the easiest to silently break.
It also makes the cost conversation honest. Once you know the failure rate and the cost of each failure type, you can compare the automated path against the manual one with real numbers rather than vendor projections. We have used exactly this exercise to advise clients against building something, which is usually the most valuable output of a discovery phase.
The broader governance frameworks point the same direction. The NIST AI Risk Management Framework treats measurement as one of its core functions, and open tooling such as OpenAI's evals repository exists precisely because evaluation is the part teams keep skipping.
Where the approach breaks down
We should be straightforward about the costs. Eval sets go stale. Customer behaviour shifts, product terminology changes, a new document format arrives from a supplier, and a set built in January quietly stops representing September. We schedule a refresh rather than pretending it is a one-time artefact.
There is also a real risk of optimising for the set instead of the problem. If the same fifty examples drive every prompt change for a year, you will eventually tune to them specifically. Holding back a portion of examples that are never used during development helps, as does adding fresh production cases on each refresh.
And some work genuinely does not justify it. A one-off internal script that a single person runs and eyeballs does not need a harness. The threshold we use is roughly: does this run without a human reading every output, and does a wrong answer reach a customer, a ledger, or a legal document? If yes, it gets an eval set before it gets a release date.
Starting from nothing
If you have an AI feature in production today with no evaluation behind it, the first version does not need to be sophisticated. Collect thirty to fifty real inputs from the last month. Write down, for each, what the correct behaviour is — a plain spreadsheet is fine. Run your current system against them and count the failures by category. That afternoon of work usually surfaces at least one problem the team did not know it had, and from there the habit builds itself.