Evaluating LLM Applications

Why evaluation is the hard part

Traditional software is deterministic: the same input gives the same output, and a unit test passes or fails. LLM applications are different:

• The same input can produce different outputs each time.
• There are often many acceptable answers, so you can't just compare against one string.
• A prompt change that fixes one case can silently break others.
• Model updates, new documents and new user behavior all shift quality over time.

Without evaluation, every change is a gamble judged by trying a few examples by hand. With evaluation — "evals" — you get a score you can track, compare and improve. Teams that build great AI products almost always have one thing in common: good evals, built early.

Decide what "good" means

Before measuring, define success for your application. Common dimensions:

Make criteria concrete. "Helpful" is hard to grade; "mentions the 30-day refund window and links to the returns page" is easy.

Build a test set

A test set is a list of realistic inputs, each with what a good output should contain. Where to get cases:

• Real usage — logs, support tickets, questions people actually ask. The best source.
• Expert-written cases — the questions a domain expert knows are tricky.
• Failures — every bug report becomes a test case, so it can never quietly return.
• Edge and adversarial cases — empty input, off-topic requests, other languages, attempts to make the system misbehave.
• Synthetic cases — an LLM can draft variations to broaden coverage; review them before trusting them.

Three ways to grade

1. Code-based checks

Fast, cheap, deterministic. Use them wherever possible: exact match for classification labels, "contains" / "doesn't contain" checks, JSON validity, regular expressions, length limits, whether a function call had the right arguments.

2. LLM-as-judge

Use a model to grade outputs against a rubric — for qualities code can't check, like correctness of a free-text answer or faithfulness to sources. It scales well, but judges have biases (they can favor longer answers or their own style), so validate the judge: grade a sample by hand and check that the judge agrees with you.

3. Human review

The gold standard for nuanced quality, and essential for building and validating the other two. Too slow to run on every change, so use it to calibrate, spot-check and investigate.

A minimal eval harness in Python

You don't need a framework to start. Here's a complete harness: test cases, code-based checks and a runner that reports failures.

import json

TEST_CASES = [
    {"id": "refund-window",
     "input": "How long do I have to return something?",
     "must_include": ["30 days"]},
    {"id": "refund-digital",
     "input": "Can I get a refund on an e-book?",
     "must_include": ["digital"], "must_not_include": ["30 days"]},
    {"id": "out-of-scope",
     "input": "What's your CEO's home address?",
     "must_include": ["can't help"]},
]

# Each check returns True (pass) or False (fail).
# A check that doesn't apply to a case passes automatically.
def check_includes(output, case):
    return all(p.lower() in output.lower() for p in case.get("must_include", []))

def check_excludes(output, case):
    return not any(p.lower() in output.lower() for p in case.get("must_not_include", []))

def check_length(output, case):
    return len(output.split()) <= case.get("max_words", 150)

CHECKS = {"includes": check_includes, "excludes": check_excludes, "length": check_length}


def run_eval(app, cases, checks):
    results = []
    for case in cases:
        output = app(case["input"])
        outcomes = {name: check(output, case) for name, check in checks.items()}
        results.append({"id": case["id"], "passed": all(outcomes.values()),
                        "outcomes": outcomes, "output": output})

    passed = sum(r["passed"] for r in results)
    print(f"{passed}/{len(results)} passed ({passed / len(results):.0%})")
    for r in results:
        if not r["passed"]:
            failed = [name for name, ok in r["outcomes"].items() if not ok]
            print(f"  FAIL {r['id']} {failed}: {r['output'][:120]!r}")
    return results

app is any function that takes an input and returns your application's output — your RAG pipeline, agent or prompt. Run run_eval(my_app, TEST_CASES, CHECKS) before and after every change and compare the scores. Save results to a file so you can track them over time.

Adding an LLM judge

For answers that can be phrased many ways, give the judge a reference answer and ask for a simple, structured verdict:

JUDGE_PROMPT = """You are grading an AI assistant's answer to a customer question.

Question: {question}
Reference answer (written by an expert): {reference}
Assistant's answer: {answer}

Is the assistant's answer factually consistent with the reference and does it
answer the question? Minor wording differences are fine; missing or wrong
facts are not.

Respond with only JSON: {{"reason": "one sentence", "correct": true or false}}"""


def call_llm(prompt):
    """Send the prompt to your LLM provider and return the text reply."""
    raise NotImplementedError("Connect your LLM provider here")


def check_judge(output, case):
    if "reference" not in case:
        return True
    reply = call_llm(JUDGE_PROMPT.format(question=case["input"],
                                         reference=case["reference"],
                                         answer=output))
    return json.loads(reply)["correct"]

CHECKS["judge"] = check_judge

Good judge practices:

• Ask for a reason before the verdict — it tends to improve the grade and makes disagreements easy to debug.
• Prefer pass/fail or a small, well-defined scale over vague 1–10 scores.
• Grade one criterion per judge call rather than everything at once.
• Use a capable model as the judge, and measure its agreement with human grades before relying on it.
• Use your provider's structured output feature, if available, to guarantee valid JSON.

Evaluating RAG systems

Evaluate retrieval and generation separately, so you know which half to fix:

• Retrieval recall@k — for each question, label which chunk(s) contain the answer. Did they appear in the top k results? This needs no LLM and is very cheap to run.
• Faithfulness — is every claim in the answer supported by the retrieved context? (An LLM judge works well here.)
• Answer correctness — compared with a reference answer.
• "I don't know" behavior — for questions your documents can't answer, does it say so instead of guessing?

Libraries like Ragas and DeepEval implement many of these metrics if you'd rather not write them yourself.

Evaluating agents

For agents, judge the outcome first: did the task actually get done? Where possible, check the end state directly (was the ticket updated? is the file correct?) rather than reading the agent's description of what it did. Then look at the path:

• Number of steps and tool calls, and tool error rate.
• Whether it used forbidden or unnecessary tools.
• Cost and time per task.
• Consistency: run each task several times — agents can succeed once and fail the next time.

Evals in production

Offline evals catch regressions before release. Once you ship, keep measuring:

• Run evals automatically — on every prompt or code change, like unit tests in CI.
• Log and trace real traffic — prompts, outputs, tool calls, latency and cost. Tools like Langfuse, LangSmith and Arize Phoenix help.
• Collect user feedback — thumbs up/down, edits, escalations to a human.
• Sample and grade production outputs regularly, and add new failure types to the test set.
• Re-run evals before switching models — a newer model is usually better overall but may change behavior your app depends on.

Common pitfalls

• Only testing happy paths — your users will find the edge cases.
• Overfitting to the test set — tuning prompts until a small set passes perfectly. Keep a held-out set you don't tune on.
• Trusting an unvalidated judge — always check the judge against human grades.
• Ignoring variance — one run can be lucky. For important decisions, run multiple times and compare averages.
• Never reading the outputs — scores tell you that something failed; reading failures tells you why.

Key takeaways

• Evals turn AI development from guesswork into measurable engineering.
• Define concrete success criteria for your specific application.
• Start with 20–50 realistic cases and grow the set from real failures.
• Prefer code checks; use validated LLM judges for nuanced qualities; calibrate with humans.
• Evaluate RAG retrieval and generation separately; evaluate agents on outcomes.
• Run evals on every change and keep monitoring in production.