PPROMPT WORKBOOK Build a prompt
Evaluation

Quality assurance

“It seems better” is a guess. A small test set turns it into a measurement, and it takes less effort than people expect.

Build the smallest test set that would embarrass you

Collect real examples of the task, including the awkward ones: the ambiguous request, the missing field, the customer who is angry, the document in the wrong format. Write down what a good answer looks like for each. That written standard is the actual work, and everything else follows from it.

How many you need depends on the difference you care about. Catching a badly broken prompt takes ten to twenty cases. Deciding whether a tweak improved things by a few percent takes hundreds. Below that, you are reading noise.

Using a model as judge, carefully

Having one model grade another's output scales well and is genuinely useful. It also has well-documented biases that will mislead you if you ignore them.

  1. Position bias is large

    Simply swapping the order of two candidate answers can flip the verdict. In one published study the reordering alone turned a loss into a win on 66 of 80 questions. Run every comparison both ways and average.

  2. Judges prefer their own voice

    Models score text that resembles their own output more highly. Use a different model to grade than the one that generated, which is also Anthropic's own published advice.

  3. Style beats substance

    Judges are harsh on unusual formatting and surprisingly forgiving of factual errors. A confident, well-formatted wrong answer often outscores a correct terse one.

  4. Use binary rubrics

    Ask for specific pass/fail criteria rather than “rate the quality from 1 to 10”. Discrete judgements against a written standard are far more stable.

Calibrate the judge before trusting it. Grade twenty examples yourself, then check whether the model agrees with you. If it does not, the judge is measuring something other than what you care about.

Public benchmarks are weak evidence

Benchmark scores are a signal, not a verdict. Accuracy on a standard benchmark can shift by several percent purely from formatting changes, by Anthropic's own account. Contamination is hard to detect and can be effectively hidden by later training stages. And a peer-reviewed analysis has alleged that a leading public leaderboard is gameable through privileged data access, a claim the leaderboard's operators publicly dispute.

None of that makes benchmarks useless. It makes your own task-representative test set more valuable than any published number.

Sources