Mmanish/karki
← All projects
Eval Tooling Work

prompt-bench

A small evaluation harness for comparing prompt variants — it runs the same test set across every variant, scores them on accuracy, cost, and p95 latency, and writes a report I can paste into a Slack thread.

The problem

At my internship, we kept ending up in arguments that went like this: someone changed a system prompt, the answers felt better in a handful of manual tests, and we shipped it. Two days later we'd notice a regression on a different kind of input and have to roll back. Then we'd argue about which version was actually better. We had no way to tell.

What we needed was a fixed set of inputs and expected behaviours, and a way to run any prompt variant across that set and compare results. A test suite. For prompts. There are plenty of commercial tools that do this. We didn't have budget for any of them, and I wanted to understand the moving parts anyway.

What I built

Three pieces, each as small as I could get away with:

  1. A cases.yaml file with inputs, expected outputs, and category tags.
  2. A runner that takes a prompt template, fills it in, hits whatever model you configured, and grades the response.
  3. A Streamlit page that diffs two runs side by side — per-case, per-category, and aggregate.

The grader is where the actual work is. For deterministic cases it's a simple equality check or regex. For open-ended ones it's an LLM-as-judge with a rubric, and I sample 10% of those for manual review. The eval set itself is stored in version control so reviewers can argue with the ground truth, not just the model. 

# a case
- id: refund_yes
  input: |
    Customer wrote: "I changed my mind, I want my money back."
  expect:
    intent: refund_request
    sentiment: neutral
  tags: [refund, easy]

What I learned from running it

  • The biggest accuracy wins came from changes I would never have made by gut feel. The clearest example: replacing "be concise" with "answer in at most two sentences" cost us 4 points of accuracy because the model started cutting off mid-reasoning. We'd never have caught that without the eval.
  • Cost matters more than I thought. One refactor saved 60% of input tokens with no accuracy loss — invisible until you put dollar signs next to each variant.
  • Per-category breakdowns saved us at least three times from shipping changes that improved aggregate score but tanked a specific kind of input.
what surprised me

Once we had numbers, the social dynamics of the team changed. Arguments about prompts stopped being arguments about taste and started being arguments about which case the model was failing on, and why. Much shorter, much more useful.

What I'd do differently

  • Build versioning of prompts into the tool from day one. We bolted it on later and lost a week of comparisons we couldn't reconstruct.
  • Make the LLM-judge prompt itself part of the eval. We tweaked it once mid-stream and didn't realize for a week that some of our "improvements" were judge drift.
  • Track latency p95, not mean. Mean hides the long tail that actually breaks user trust.