jd five

Evaluating Agents

You changed a prompt. Feels better. Is it?

Evaluating an agent means: run fixed cases, score the outputs, compare before vs after.

Guessing in the chat window is not evaluation.

Tiny eval loop

  1. Keep a list of cases (input + what “good” means)
  2. Run version A (baseline) → score
  3. Run version B (your change) → score
  4. Keep B only if the score is better (or equal and clearer)
Same cases. Two versions. Compare.
  Cases (fixed)
       │
       ├──────────► Agent v1 ──► score 1/3
       │
       └──────────► Agent v2 ──► score 3/3

What to score (start simple)

For this course lab, “good” means: the reply contains required words (order id, refunded / refused, etc.).

Later you can use an LLM-as-judge. Start with rules you can explain.

Case Ask Good if reply mentions…
1 Refund 4412 4412 and refund
2 Status of 5501 5501 and human or approval
3 Hello help or support

v1 is a weak stub. v2 is a better stub. The point is the harness, not a fancy model.

Words to keep

Word Meaning
Eval A test run that scores behavior
Baseline The version you compare against
Regression A case that used to pass and now fails

What you should remember

  1. Fixed cases beat vibes.
  2. One number (pass rate) makes “better” visible.
  3. Re-run the suite when you change prompts, tools, or guards.

See it in Code

Open Code. Watch baseline vs v2 pass rates. No API key required.