_(15515758528).jpg){kind=link}
Every team building on language models hits the same wall. The feature seems better after a change, but nobody can prove it, so decisions get made on impressions and the quality of the product drifts wherever the last opinion pushed it.
Evaluation is how you get out of that. It is also the part everyone postpones, because it feels like overhead until the week it becomes the only thing keeping the feature alive.
Start with cases, not metrics
Before choosing a scoring method, collect inputs. Real ones.
Pull them from actual usage if you have it, from support tickets if you do not, and from the edge cases you already know are awkward. A hundred well chosen cases beats ten thousand generated ones, because the value is in coverage of the strange, not volume of the ordinary.
Every case needs to record what a good answer looks like. Not necessarily the exact words, but the facts it must contain, the things it must not claim, and the shape it should take.
Score the cheapest thing that works
Not everything needs a sophisticated judge. Work up the ladder and stop early.
Deterministic checks catch more than people expect. Is it valid JSON. Does it contain the required field. Is it under the length limit. Does it cite a document that actually exists. Did it avoid the seven phrases we have banned. These are free, instant, and they catch a large share of real regressions.
Reference comparison works where there is a known answer. Exact match for classification, or a similarity score for short factual responses.
A model as judge is for the genuinely subjective cases: is this summary faithful, is this tone right, is this answer actually responsive to the question. It is useful and it is not free. A judge needs its own validation, because a judge that disagrees with your human reviewers is just an expensive random number.
Human review stays the ground truth. You do not need much of it, but you need some, and you need it regularly enough to notice when the automated scores have stopped correlating with what people think.
Measure the failures you actually care about
Generic quality scores hide the problems that matter. Break the score apart.
For a retrieval system I want to know separately: did it find the right source, did it use the source it found, did it make anything up, and did it correctly refuse when the answer was not there. A single blended number can stay flat while the fabrication rate doubles.
The refusal case deserves its own attention. Almost every evaluation set is full of answerable questions, so almost every system is untested on the one behaviour that causes the most damage when it goes wrong.
Run it on every change
An evaluation you run occasionally is a research project. An evaluation that runs on every pull request is a safety net.
It does not have to be slow or expensive. A fast tier of deterministic checks on every commit, and the full set including the judge before release, gets you most of the value for very little time.
The number that matters is the difference
Absolute scores are close to meaningless. Eighty two percent on your own evaluation set tells you nothing about anything.
What matters is the direction of travel between two versions, on a set that has not changed. That is the only question evaluation can genuinely answer, and it is the only one you need it to.
Why this is the whole discipline
Ordinary software testing asks whether the code does what it says. Evaluation asks whether the system is getting better or worse, on a component that will never be exactly right.
That is a different question, it needs different tools, and it is the skill that separates AI features that hold up from AI features that quietly rot.