For most of my first year working with LLMs, I shipped prompt changes the same way I'd shipped HTML changes in the early days: I'd write a thing, squint at the output, decide it looked better, and commit. There were no tests, because what would the tests even look like. There were no metrics, because the output was prose. There were arguments on the team about whether the new version was better, because everyone had different examples in their head.
We were vibe-coding. It worked until it didn't. About six months in I watched us roll back three prompt changes in a single week, each one having broken something that a different person had only just fixed. We'd been improving the same prompt against three different mental models of what "improvement" meant. It was, I came to understand, not a prompt problem. It was an epistemics problem. We didn't know what good looked like.
The thing nobody tells you
The thing nobody tells you when you start working with LLMs is how much of the work is not about the model. It is about deciding, in advance and in writing, what "good" means — clearly enough that two people can grade the same output and agree. The model is the easy part. Knowing what to ask of it is the hard part.
A prompt evaluation is, at its smallest, a list of inputs and the answers you wanted. That's it. You run the prompt against each input, you grade the answer against what you wanted, and you have a number. The number isn't the truth — no number is the truth — but it is a coordinate. Two coordinates make a direction. And a direction is what you need.
Without a test set, you're not iterating on a prompt — you're rearranging deck chairs.
What the eval set looks like
Mine is a YAML file. Thirty to a hundred cases, depending on the project. Each case has an input, an expected output (or a rubric, for open-ended questions), and a tag or two for categorization. Boring. Stable. Lives in version control next to the code.
# cases.yaml
- id: refund_explicit
input: |
Customer wrote: "I changed my mind, I want my money back."
expect:
intent: refund_request
tags: [refund, easy]
- id: refund_implicit
input: |
Customer wrote: "I'd rather not have this anymore, can we work
something out?"
expect:
intent: refund_request
tags: [refund, hard]
Two things matter more than they look like they do. Tags — because almost every regression I've ever caught has been a category regression that the aggregate score hid. Hard cases mixed with easy ones — because if your eval set is all easy cases, every variant will look fine.
How to grade open-ended outputs
For deterministic cases — intent classification, structured extraction, anything you can express as equality or a regex — the grader writes itself. For open-ended outputs (summaries, customer-facing replies, explanations), I use an LLM-as-judge with a rubric, and I sample ten percent for manual review.
This sounds dirty and it is dirty, but it works if you treat the judge prompt itself as part of the system under test. Version it. Run your candidate variant against your locked-in judge from last week, not against a judge prompt you also just edited. Otherwise — and I learned this the hard way — half of your "improvements" will turn out to be the judge being more lenient.
Track three numbers, not one
For every variant I run, I keep three numbers per case:
- Accuracy. Did the answer match the rubric?
- Cost. Input tokens × input price + output tokens × output price.
- Latency. p95, not mean. Mean hides the long tail that breaks user trust.
The biggest accuracy wins in my career so far have come from changes I would never have made on gut feel. The clearest example: I replaced "be concise" with "answer in at most two sentences" in a customer-service prompt. Accuracy dropped four points because the model started cutting itself off mid-reasoning. We would never have caught that without the eval.
Cost is the number I underestimated longest. One refactor of a context builder saved 60% of input tokens with no accuracy loss. Without putting dollar signs next to each variant, that change would have looked equivalent. With them, it was an obvious yes.
What it changes about the team
The least technical thing I want to say in this essay is the most important. Once we had numbers, the social dynamics of the team changed. Arguments about prompts stopped being arguments about taste — which never end — and started being arguments about which case the model was failing on, and why. Those arguments are much shorter, and much more useful.
We stopped having weekly fights about whether the new prompt was better. We started having weekly meetings about whether the eval set was still representative. Both are productive conversations. Only the second one ends with a smaller backlog.
What I'd tell past me
- Build the eval before the prompt. Always. The eval is the spec.
- Write thirty cases before you ship anything. Then twenty more.
- Tag every case. You'll thank yourself the first time aggregate goes up and one category quietly tanks.
- Version your judge prompt. Drift is silent and expensive.
- p95 latency, not mean. The long tail is where the user gives up.
- Track cost in dollars, not tokens. Tokens lie about what you're spending.
None of this is exciting. None of this is the part of AI that gets you invited to give a talk. But every team I've watched ship LLM features that actually work, when I've gone looking, has a boring spreadsheet of cases somewhere in their repo. The boring spreadsheet is the work. The prompt is the easy bit. Build the spreadsheet first.