Evaluating & Testing Prompts

Reviewed & published by Brayan K

Evaluating prompts means measuring their quality instead of guessing. You build an eval set of inputs paired with expected outcomes, score your prompt against it, and use the numbers to improve.

Part of the free Prompt Engineering course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

This lesson covers exact-match, rubric, and LLM-as-judge grading, regression testing, A/B comparison, and the iteration loop that turns a good prompt into a great one.

What You'll Learn in This Lesson

• Building an eval set of inputs and expected outcomes
• Exact-match vs rubric vs LLM-as-judge grading
• Regression-testing prompts after a change
• A/B comparing two prompt versions
• Metrics that quantify prompt quality
• The measure, adjust, re-measure iteration loop

1 Build an eval set

An eval set is a list of test inputs paired with the outcomes you expect. Run your prompt against it and score the results:

2 Three ways to grade

Pick the method that fits how open-ended the task is. Many real systems combine more than one.

3 Regression test and A/B compare

A change that fixes one case can quietly break another. Re-run your eval set after every edit, and compare versions head to head:

Try it for real

Everything above is advice until you can score it. Here is the whole loop in plain Python: an eval set, two prompt versions with their recorded replies, and a deterministic grader. Run it and watch one sentence added to a prompt move the score from 0/2 to 2/2 — that measured difference is what an eval set buys you.

# A prompt eval harness. No model calls: the four replies below are recorded
# samples, pasted in so the grading logic is what you study and change.

# 1) THE EVAL SET — the questions you care about, each with a rule for passing.
eval_set = [
    {"q": "What is our refund window?",      "must_include": ["30 days"],       "must_not": ["I think", "probably"]},
    {"q": "Do you cover postage on faults?", "must_include": ["we pay", "postage"], "must_not": ["maybe"]},
]

# 2) TWO PROMPT VERSIONS, and what each actually replied. Version B adds
#    "Answer only from the policy text and never speculate."
replies = {
    "A (no grounding rule)": [
        "I think it's probably around 30 days, but check with support.",
        "Maybe — postage is sometimes covered.",
    ],
    "B (grounding rule added)": [
        "Refunds are available within 30 days of delivery.",
        "Yes. For faulty goods we pay the postage.",
    ],
}

# 3) THE GRADER — deterministic, so the score means the same thing every run.
def grade(reply, case):
    reply_l = reply.lower()
    missing = [p for p in case["must_include"] if p.lower() not in reply_l]
    hedged  = [p for p in case["must_not"]    if p.lower() in reply_l]
    return (not missing and not hedged), missing, hedged

# 4) SCORE EACH VERSION
for version, answers in replies.items():
    passed = 0
    print(f"--- {version}")
    for case, reply in zip(eval_set, answers):
        ok, missing, hedged = grade(reply, case)
        passed += ok
        flag = "PASS" if ok else "FAIL"
        detail = ""
        if missing: detail += f"  missing={missing}"
        if hedged:  detail += f"  hedging={hedged}"
        print(f"  {flag}  {case['q']}{detail}")
    print(f"  score: {passed}/{len(eval_set)}")
    print()

print("The rule you added is worth one sentence in the prompt and moved the")
print("score from 0/2 to 2/2. Without an eval set you would be guessing.")

# 🎯 YOUR TURN — finish the grader, then read the score it produces.

eval_set = [
    {"q": "When does an annual plan renew?", "must_include": ["automatically"], "must_not": ["I guess"]},
    {"q": "Where is my receipt sent?",       "must_include": ["billing address"], "must_not": ["not sure"]},
    {"q": "Can I cancel today?",             "must_include": ["account page"],   "must_not": ["perhaps"]},
]

# Recorded replies from one prompt version. One of them hedges.
replies = [
    "Annual plans renew automatically each year.",
    "I guess it goes to your email somewhere.",
    "Yes — cancel any time from the account page.",
]

def grade(reply, case):
    reply_l = reply.lower()
    # 1) A phrase is missing if it does NOT appear in the reply.
    missing = [p for p in case["must_include"] if p.lower() ___ reply_l]
    # 👉 replace ___ with  not in

    # 2) A reply hedges if a banned phrase DOES appear.
    hedged  = [p for p in case["must_not"] if p.lower() ___ reply_l]
    # 👉 replace ___ with  in

    # 3) It passes only when both lists are empty.
    return (not missing) ___ (not hedged)
    # 👉 replace ___ with  and

passed = sum(grade(r, c) for r, c in zip(replies, eval_set))

print(f"score: {passed}/{len(eval_set)}")
for case, reply in zip(eval_set, replies):
    print(("PASS  " if grade(reply, case) else "FAIL  ") + case["q"])

# ✅ Expected output:
# score: 2/3
# PASS  When does an annual plan renew?
# FAIL  Where is my receipt sent?
# PASS  Can I cancel today?

Common Mistakes (and the fix)

• Judging a prompt by one example. — Build a proper eval set of many cases.
• No grading method. — Choose exact match, a rubric, or LLM-as-judge.
• Not re-testing after edits. — Regression-test the full set every time.
• Guessing which version is better. — A/B compare on the same inputs.

⏱ Test Yourself — Timed Quiz

10 quick questions, 12 seconds each. Instant feedback — beat the clock!

🎉 Lesson Complete

• ✅ Build an eval set of inputs and expected outcomes
• ✅ Grade with exact match, a rubric, or LLM-as-judge
• ✅ Regression-test the full set after every change
• ✅ A/B compare prompt versions on the same inputs
• ✅ Iterate: measure, adjust, and keep the higher score

Practice quiz

Continue this course

• Previous: Prompt Injection & Safety

Frequently asked questions

What does it mean to evaluate a prompt?

Evaluating a prompt means measuring how well it does its job instead of guessing. You build an eval set: a collection of test inputs paired with the outcomes you expect. Then you run your prompt against those inputs and score the results. This turns 'I think this prompt is better' into 'this prompt scores 92 percent versus 85 percent,' which is far more reliable.

What are the ways to grade outputs?

Three common approaches. Exact-match grading compares the output to one correct answer, ideal when there is a single right result. Rubric grading scores open-ended outputs against defined criteria, like clarity or completeness. LLM-as-judge uses another AI to score outputs against your rubric at scale. You pick the method that fits how open-ended the task is.

Why regression-test prompts when I change them?

A change that fixes one case can quietly break another. Regression testing means re-running your whole eval set after every edit so you catch new failures immediately. It is the same idea as software tests: keep a stable set of cases, and make sure each change improves the score without breaking what already worked.

How do A/B testing and iteration fit together?

A/B comparison runs two prompt versions on the same eval set and tells you which one scores higher, removing the guesswork from 'which wording is better.' Iteration is the loop around it: measure, adjust the prompt, measure again, and keep the version that wins. Over several rounds, this steady measure-and-improve cycle produces prompts you can trust.