Iteration and Evaluation: Small Test Sets, A/B Testing, Version Control, Common Anti-Patterns

Today's Goals

1. Build a test set of about ten cases for a prompt, and state what each case is testing
2. Write an A/B evaluation script that compares the pass rate of two prompt versions on the same inputs
3. Version-control a prompt, and explain how to recognize five common anti-patterns

For three days we have been editing prompts while dodging one question: after you edit, how do you know it got better? Two tries in a chat window that feel fine can turn into two hundred failures out of a thousand production runs. Today we treat the prompt as code — with tests, with comparisons, with a version number. When you have read the walkthrough and finished the lab, come back to the top of the page and check off the three goals.

Plain-Language Walkthrough

Why editing prompts by feel makes them worse over time

The colleague follows the ticket, you tack on one more line, and a week later you discover three other tasks that used to go fine are now breaking — because that one new line conflicts with them, and you only ever checked "this one task." The first round went wrong, you added a sentence, they redid it, and this time it was right. You were pleased. You did not notice the collateral damage, because each time you looked at one task only.

Iterating on a prompt by feel is that same process, sped up. You see one input come out wrong, change a sentence, rerun that one input, see it pass, and call it done. But a model's response to a prompt is global: the rule you added to fix one edge case changes its behavior on every input. Without a fixed set of inputs you rerun in full every time, you will forever see only the case you just fixed and never the cases you just broke. So the prompt gets longer, its behavior gets less predictable, and eventually nobody dares touch it.

Software solved this problem long ago: have tests before the change, run them all after the change, and a red one tells you what broke. What prompts lack is exactly that — a baseline. Without a baseline, "better" and "worse" are both feelings. With one, "this version went from six of ten to nine of ten, but the tenth used to pass and now fails" is a fact you can discuss and decide on. Building that baseline is today's job, and it does not have to be large. Ten cases are enough to start.

A small test set: are ten cases enough, and what should each look like

Start with whether ten is enough. For a mature product, obviously not. But for the step from editing by feel to editing on evidence, ten already filters out the overwhelming majority of blind changes. The reason is that prompt failures are highly concentrated: models almost never get normal inputs wrong, and all the failures live at the edges and in the adversarial cases. So the value of ten cases is not in the count, it is in the distribution.

A good small test set is roughly three or four cases in each of three categories. Normal inputs, written conventionally, which both versions should get right. Their job is to hold the floor — if a new version breaks these, that is a serious regression. Edge inputs, where the requirement omits a default, a field is optional, the path carries a parameter, or the phrasing is loose ("the create endpoint for todos" instead of POST /todos). These test whether the prompt spells out its default rules. Adversarial inputs, carrying distractors (a paragraph about a production incident before the actual requirement), a change of mind mid-sentence ("endpoint A — no wait, B"), or a smuggled extra demand ("and make the response paginated while you are at it"). These test whether the prompt keeps the model on the point.

Each case has three parts: the input, a human-labeled expected answer, and a category tag. The expected answer is human work, and there is no shortcut — label one case wrong and the whole evaluation is skewed, and you will think the prompt is at fault and keep editing the prompt. After writing each case, reread the input and confirm the answer is unique. Using the task that runs through the course, an edge case looks like this. It reuses D3's extraction task and labels only the four key fields:

{
  "id": "c04",
  "kind": "edge",
  "input": "POST /todos, add validation: title is required, at most 200 characters. (The requirement says nothing about what a failure returns.)",
  "expected": { "endpoint": "/todos", "method": "POST", "failureStatus": 400, "fields": ["title"] }
}

It tests one very specific thing: when the requirement omits a status code, does the model apply the team default of 400, or invent a 422? The "samples the model could not possibly know" from D2 belong here too, one or two of them, to see whether it honestly says so or manufactures an answer.

How to decide what counts as right: exact match, field comparison, and LLM-as-judge

With inputs and expected answers in hand, the next question is what counts as right. How tight you make this decides whether the evaluation means anything: too tight and you count phrasing differences as errors, too loose and you let real errors through. Three approaches, each with its own range.

Exact match: the serialized output equals the expected answer exactly. Suitable only when the output is a single value, such as a classification label. For structured output it is far too tight — the model writes post instead of POST, or reorders the field array, and you score it wrong even though the meaning is identical, sending you off to fix a problem that does not exist.

Field comparison: compare field by field, each field with its own rule. Method ignores case; path ignores a trailing slash and tolerates a missing leading slash; the field list is deduplicated and compared ignoring order and case; the status code must match exactly. The principle is compare the fields you care about and tolerate the differences you do not. This is the standard for structured output, and it is the biggest payoff of D3 turning the output into fixed fields — with free text, this step is simply impossible. The scoring function should return a list of mismatched field names, not a boolean, because when you aggregate you need to know what went wrong: if four of ten failures are all failureStatus, the problem is unmistakably that the default was never stated.

const norm = (s: string): string => s.trim().toLowerCase()
const normPath = (s: string): string => {
  const p = norm(s).replace(/\/+$/, '')
  return p.startsWith('/') ? p : `/${p}` // a missing leading slash is phrasing, not misunderstanding
}
 
// Returns the list of mismatched field names; an empty array means pass.
// One layer more information than a boolean: it tells you what went wrong.
function score(actual: Record<string, unknown>, expected: Expected): string[] {
  const diffs: string[] = []
  if (typeof actual.endpoint !== 'string' || normPath(actual.endpoint) !== normPath(expected.endpoint)) diffs.push('endpoint')
  if (typeof actual.method !== 'string' || norm(actual.method) !== norm(expected.method)) diffs.push('method')
  if (actual.failureStatus !== expected.failureStatus) diffs.push('failureStatus') // status codes get no tolerance
  const got = Array.isArray(actual.fields) ? [...new Set(actual.fields.map((f) => norm(String(f))))].sort() : null
  const want = [...new Set(expected.fields.map(norm))].sort()
  if (!got || got.length !== want.length || got.some((f, i) => f !== want[i])) diffs.push('fields')
  return diffs
}

def norm(s: str) -> str:
    return s.strip().lower()
 
 
def norm_path(s: str) -> str:
    p = norm(s).rstrip("/")
    return p if p.startswith("/") else f"/{p}"  # a missing leading slash is phrasing, not misunderstanding
 
 
# Returns the list of mismatched field names; an empty list means pass.
# One layer more information than a boolean: it tells you what went wrong.
def score(actual: dict, expected: dict) -> list[str]:
    diffs: list[str] = []
    if not isinstance(actual.get("endpoint"), str) or norm_path(actual["endpoint"]) != norm_path(expected["endpoint"]):
        diffs.append("endpoint")
    if not isinstance(actual.get("method"), str) or norm(actual["method"]) != norm(expected["method"]):
        diffs.append("method")
    if actual.get("failureStatus") != expected["failureStatus"]:
        diffs.append("failureStatus")  # status codes get no tolerance
    got = sorted({norm(str(f)) for f in actual["fields"]}) if isinstance(actual.get("fields"), list) else None
    want = sorted({norm(f) for f in expected["fields"]})
    if got != want:
        diffs.append("fields")
    return diffs

LLM-as-judge: when the output is free text — a summary, an email — there are no fields to compare, so you have another model score it against rubric you write. It covers tasks the first two approaches cannot touch, and the cost is real: the judge itself makes mistakes, it leans toward longer and prettier answers, and a vague rubric produces arbitrary scores. Using it presupposes that the rubric is as concrete as a work ticket ("does the summary include the status code, the field names, and the failure behavior — one point each"), and that you periodically sample the judge's verdicts by hand. The rule of thumb: if field comparison can do the job, do not reach for a judge; where a judge is unavoidable, calibrate it first against ten samples a human has already scored.

A/B comparison: one set of inputs, two prompt versions, one table

With scoring in place, A/B is just a loop: for each input in the test set, run the extraction once with version A's prompt and once with version B's, score each, and write both into the same table. Each row is one case, the two columns are the two versions, passes get a check, and failures name the field that broke.

case  kind         v1                        v2
c01   normal       PASS                      PASS
c04   edge         FAIL failureStatus        PASS
c09   adversarial  FAIL endpoint             PASS
c10   adversarial  PASS                      FAIL failureStatus
 
Pass rate: v1 6/10 (60%)   v2 9/10 (90%)
Fixed by v2:  c04, c05, c07, c09
Broken by v2: c10   <- these go in the CHANGELOG under known regressions

The three summary lines below the table matter in the opposite order from most people's intuition. Pass rate matters least — it only gives you a total. Fixed tells you the change actually solved the problem you were aiming at. Broken matters most: it tells you what regression the new version introduced. In the example above, v2 goes from six to nine and looks like a clear win, but c10 used to pass and now fails — v2 added a rule saying "use 400 when no status code is given," the model took it too literally, and it now rewrites an explicitly requested 422 into 400. Without this table, that regression walks quietly into production.

Two engineering details. First, the same set of inputs: both versions must run identical cases, and changing the cases makes the comparison meaningless. Second, randomness: a real model can give two different answers to the same input, so for a serious evaluation run each case three times and take the majority, or set the temperature to 0. For offline practice, fixed mock outputs are fine. Today's lab uses rules in offline mode to simulate which edges v1 gets wrong and what v2 fixes and breaks, so you can see the whole picture without an API key.

Version control: a prompt is code, so it belongs in the repo with a changelog

By now the prompt has every characteristic of code: it is a function with parameters (D3), it has a test set, and it has an evaluation script that produces a pass rate. So manage it like code: in the repository, with a version number and a changelog.

The mechanics are simple. The prompt body lives in its own file, rather than scattered across string concatenation in business code, and each version has an id. Next to it sits a changelog, and each entry records four things: what changed, why (which failing tests it targets), the pass rate that came out, and the known regressions.

## v2 (current)
- Change: added five rules - default status 400, uppercase method and normalized path,
  keep field names verbatim, a change of mind means the last one wins, ignore smuggled extras
- Reason: v1 failed c04 (guessed 422 with no status given), c05 (mangled a colloquial path),
  c07 (missed pageSize), c09 (honored only the first of two conflicting endpoints)
- Result: v1 6/10 -> v2 9/10
- Known regression: c10 - "no status means 400" was learned too literally, so an explicitly
  requested 422 gets rewritten to 400. Next version restates it as "use the requirement's own
  status when it gives one" and moves it to the top of the rule list

One difference from code version control is worth calling out. Code changes are usually local: editing one function does not affect another. Prompt changes are global: adding one sentence can change behavior on every input. So a prompt changelog treats known regressions as a mandatory field, whereas a commit message usually has no such field. The other difference is the cost of rollback: rolling a prompt back is swapping one string, essentially free, which makes "roll back to the previous version first, investigate afterward" an even more obvious response to a production problem than it is with code — provided you have a version number to roll back to.

Five common anti-patterns

Finally, collect the traps mentioned across the past three days into one list. Each comes with a way to recognize it, because the hard part of an anti-pattern is not fixing it but spotting it.

Growth by accretion. Every failure earns another sentence, and six months later the prompt is two thousand words and nobody knows which lines still do anything. Recognize it: delete a paragraph and run the test set. If the pass rate is unchanged, that paragraph is dead. With a test set, deletion becomes safe for the first time.

The universal role. "You are a full-stack expert in front end, back end, security, testing, and documentation." The more complete the role, the less information it carries, and the model has no idea what you want it to focus on. Recognize it: can you delete half the role description without affecting a single test case?

Negative instruction pileup. "No pleasantries," "no explanations," "no Markdown," stacked in a row. Models follow bare negatives unreliably, and a negative also drags the word into context (writing "do not mention competitors" sometimes makes it mention them). Recognize it: can each negative be rewritten as a positive with a reason ("start directly with the first section, because the output is parsed section by section")?

Contaminated examples. From D2: the example's format conflicts with the description, the example labels are skewed, or an example contains "and so on." Recognize it: pull the output halves of the examples out on their own and compare them word for word against the format box.

Testing once. Try one case in a chat window after the edit and ship it. This one needs no recognition method — the entire day was about it.

Source Reading

Hands-On Lab

All five criteria of this lab are doable without an API key. Open labs/prompt-engineering-5days/day-04-prompt-ab-eval, run solution/ first to see the full comparison table, then go back to starter/:

1. In solution/, run MOCK=1 pnpm start. Study the three-category distribution across the ten cases, which cases each of v1 and v2 fails, and the fixed and broken lines.
2. Switch to starter/ and run once. Notice that there are only 4 cases, that v1 fails all four (because scoring is too tight), and that the summary is placeholder text — those three spots are the three code exercises.
3. Extend src/cases.ts to ten: 3 edge cases (colloquial phrasing, a PUT full update, GET query parameters) and 3 adversarial ones (with distractors, a change of mind, a smuggled extra demand), labeling each expected answer by hand and rereading the input to confirm the answer is unique.
4. Complete score as a field-by-field comparison (tolerating case, trailing slashes, and field order, with the status code exact), then rerun: v1 should pass 6/10 with each of the four failures naming its field.
5. Complete the summary (pass rates, fixed, broken, per-category counts) and put the numbers into the v2 entry of CHANGELOG.md. Then write a v3 that fixes c10's regression, rerun, and record it.

Interview Questions

Today's three questions are in the question bank below, covering test set design, how prompt versioning differs from code version control, and the boundaries of LLM-as-judge. The second is the most frequently asked prompt engineering question in interviews outside China; derive it yourself before reading the analysis.

Checklist and Tomorrow

• Build a test set of about ten cases for a prompt, and state what each case is testing
• Write an A/B evaluation script that compares the pass rate of two prompt versions on the same inputs
• Version-control a prompt, and explain how to recognize five common anti-patterns
• All 5 acceptance criteria of the A/B evaluation lab pass
• You can answer at least 2 of the 3 interview questions without looking at the points

Tomorrow (D5) is the last day of the course, and it answers a question you will hit sooner or later: when the same prompt moves from one model to another, why does it break, exactly where does it break, and how do you organize a system prompt so that moving it does not hurt. The test set you built today goes straight to work — run it after the migration and the pass rate is your answer on whether anything regressed. D5 also closes with what to study next, the Claude course or the Codex course, and how to choose.