Building an Evaluation Dataset

An evaluation dataset is the set of prompts, inputs, references, rubrics, and metadata you use to decide whether a generative AI system is getting better or worse. For a production assistant, summarizer, agent, retriever, or classifier, the dataset is the practical definition of quality. It converts vague goals such as helpful, accurate, safe, concise, or on brand into repeatable examples that can be run before and after a prompt, model, retrieval, or tool change.

The outcome of this lesson is a concrete workflow for building that dataset: sample real tasks, label the task shape and risk, write rubrics before scoring, separate stable regression cases from fresh exploration cases, and preserve enough metadata to explain failures. In the Evaluation and Safety section, this is the chapter that turns judgment into engineering evidence. Later checks, dashboards, and release gates are only as useful as the examples they measure.

What The Dataset Contains

A useful generative AI evaluation row is not just a prompt. It usually contains an input, optional context documents, the expected behavior, a scoring rubric, metadata, and a split assignment. The input is what the application will send to the model or agent. The context is the evidence available to the system, such as retrieved passages, a customer record, or tool results. The expected behavior may be a reference answer, but for open-ended generation it is often a rubric that defines what must be present, what must be absent, and what would make the answer unacceptable.

The separation matters. If you mix the expected answer into retrieved context, you test memorization of the benchmark rather than behavior. If you omit metadata, a single aggregate score can hide that the system improved on easy English support questions while regressing on high-risk policy questions.

Internal Mechanics

Building the dataset starts with a task taxonomy. Instead of collecting random examples, define buckets that reflect the application: intent, channel, document type, user sophistication, language, risk level, dependency needed, and failure cost. A customer support chatbot might use refund questions, warranty questions, account access, troubleshooting, and escalation. A code assistant might use bug explanation, patch generation, test writing, API migration, and security review.

Next, sample from real traffic or realistic source material. Real tasks contain ambiguity, incomplete wording, spelling variation, adversarial phrasing, and edge cases that synthetic happy paths miss. Remove private data or replace it with faithful placeholders before adding examples to a shared evaluation set. Keep the task shape intact: replacing a customer name is usually fine, but simplifying the complaint can remove the very behavior you need to test.

Then write the rubric before running candidate systems. This prevents the common failure where the rubric is adjusted to justify a preferred answer. A good rubric contains positive requirements, negative constraints, severity, and scoring guidance. For example: cite the refund policy, state that opened digital goods are nonrefundable, offer escalation when the purchase date is missing, and do not invent a refund approval.

Finally, split the data by purpose. A small stable regression set catches previously fixed bugs. A development set guides prompt and retrieval tuning. A holdout set estimates performance on unseen examples and should be touched rarely. A canary set can contain recently observed incidents that are too important to wait for a quarterly benchmark refresh.

API And File Anatomy

The storage format can be JSONL, CSV, a database table, or an evaluation platform record. JSONL is often convenient because each line can hold nested context, rubrics, and metadata without forcing everything into flat columns. Regardless of format, treat the schema as a contract. Version it when fields change, validate it in continuous integration, and make every row executable by the evaluation runner.

Common fields include schema_version, id, task_type, input, context, expected, rubric, metadata, and split. The runner should load the row, call the system under test, score the output with deterministic checks or human/model grading, and record score, latency, cost, and failure notes. Deterministic checks are best for exact requirements such as valid JSON, no forbidden phrase, citation count, or required field presence. Rubric grading is better for open-ended reasoning and communication quality, but it needs calibration examples and periodic human review.

Example 1: Candidate Rows From Incidents

The first example turns observed failures into candidate evaluation rows. The point is not to score yet. It is to preserve the task, source, risk, and intent so the team can choose a balanced set.

from dataclasses import dataclass

@dataclass(frozen=True)
class Incident:
    id: str
    source: str
    text: str
    risk: str

@dataclass(frozen=True)
class EvalCandidate:
    id: str
    input: str
    intent: str
    source: str
    risk: str

def infer_intent(text: str) -> str:
    lowered = text.lower()
    if "refund" in lowered or "charged" in lowered:
        return "refund_or_billing"
    if "delete" in lowered or "account" in lowered:
        return "account_management"
    return "general_help"

def build_candidates(incidents: list[Incident]) -> list[EvalCandidate]:
    return [
        EvalCandidate(
            id=incident.id,
            input=incident.text.strip(),
            intent=infer_intent(incident.text),
            source=incident.source,
            risk=incident.risk,
        )
        for incident in incidents
    ]

incidents = [
    Incident("E001", "support", "I was charged twice and need a refund", "high"),
    Incident("E002", "chat", "Can you delete my old account?", "medium"),
]

for row in build_candidates(incidents):
    print(f"{row.id}: {row.risk}/{row.source}/{row.intent}")

The expected output is deterministic:

E001: high/support/refund_or_billing
E002: medium/chat/account_management

This example shows why raw sampling is only the first step. The two incidents have different risks and intents. If billing failures are costly, E001 probably belongs in the stable regression set after the rubric is written.

Example 2: Balanced Regression Split

A regression set should be stable and representative, not merely the first rows collected. This example selects one item from each intent and risk pair, then sends the remainder to a development split.

from collections import defaultdict

rows = [
    {"id": "E001", "intent": "refund", "risk": "high"},
    {"id": "E002", "intent": "account", "risk": "medium"},
    {"id": "E003", "intent": "refund", "risk": "high"},
    {"id": "E004", "intent": "troubleshooting", "risk": "low"},
]

def assign_splits(items: list[dict[str, str]]) -> dict[str, str]:
    seen: set[tuple[str, str]] = set()
    split_by_id: dict[str, str] = {}
    for item in sorted(items, key=lambda row: row["id"]):
        bucket = (item["intent"], item["risk"])
        if bucket not in seen:
            split_by_id[item["id"]] = "regression"
            seen.add(bucket)
        else:
            split_by_id[item["id"]] = "development"
    return split_by_id

for row_id, split in assign_splits(rows).items():
    print(f"{row_id}: {split}")

The expected output is:

E001: regression
E002: regression
E003: development
E004: regression

The trade-off is visible. This small rule gives broad coverage quickly, but it may underrepresent frequent production traffic. Mature teams usually combine quota-based coverage with volume-based sampling and explicit high-severity cases.

Example 3: Rubric Scoring

For open-ended answers, exact string matching is too brittle. A rubric can still include deterministic checks for required and forbidden claims.

def score_refund_answer(answer: str) -> dict[str, object]:
    normalized = answer.lower()
    required = ["refund policy", "purchase date"]
    forbidden = ["approved your refund", "guaranteed refund"]
    missing = [term for term in required if term not in normalized]
    violations = [term for term in forbidden if term in normalized]
    passed = not missing and not violations
    return {"passed": passed, "missing": missing, "violations": violations}

answer = "Please share the purchase date so I can check the refund policy."
print(score_refund_answer(answer))

The expected output is:

{'passed': True, 'missing': [], 'violations': []}

This is not a complete grader, but it is a strong guardrail. It checks that the answer asks for the missing date and grounds the response in policy, while preventing the system from claiming an approval it cannot make.

Design Choices And Trade-Offs

Real examples are more representative than synthetic examples, but they require privacy review and careful redaction. Synthetic examples are useful for rare combinations, adversarial patterns, and bootstrapping before launch, but they can overfit to the generator that created them. The best datasets usually combine both, with source metadata kept visible.

Reference answers are efficient when the output has a narrow correct form, such as extraction or classification. Rubrics are better for summarization, advice, tutoring, and agent behavior where multiple good answers exist. Human grading gives nuanced signal but is expensive and slow. Model-based grading scales better but can drift, inherit bias, and reward fluent wrong answers unless anchored with calibration rows.

Dataset size is another trade-off. A tiny set runs quickly but misses important behavior. A huge set slows iteration and can make every experiment expensive. Use layers: a fast smoke set for every change, a stable regression set for pull requests, a broader development benchmark for tuning, and a holdout set for release decisions.

Failure Modes And Troubleshooting

Security, Performance, And Reliability

Evaluation datasets often contain user-like prompts, documents, and failure transcripts, so treat them as sensitive engineering assets. Redact personal data, store access-controlled copies, and avoid placing raw production conversations in public repositories. Keep a lineage field that records whether a row came from production, a red-team exercise, a synthetic generator, or a manually written case.

Performance matters because evaluations run repeatedly. Cache immutable context, sample large suites by split, and run cheap deterministic validators before expensive model graders. Reliability depends on repeatability: pin dataset versions for release comparisons, record the application version and model configuration used for each run, and avoid silently changing rubrics between experiments.

Hands-On Lab

Prerequisites: Python installed locally, a text editor, and five to ten anonymized examples from the generative AI workflow you care about. No model API is required for this lab.

1. Create a table or JSONL file with columns for id, input, intent, risk, source, rubric, and split.
2. Add at least six examples: three common tasks, one ambiguous task, one high-risk task, and one task based on a known failure.
3. Write a rubric for each row before testing any new prompt or model. Include at least two required behaviors and one forbidden behavior.
4. Assign one example from each important bucket to regression, put tuning examples in development, and reserve at least one row as holdout.
5. Run a simple deterministic checker like the third example against one answer per row.
6. Record failures as row-level notes instead of only keeping an aggregate score.

Verification: every row should have a stable identifier, a nonempty rubric, a split, and metadata that explains why it was included. At least one row should fail when given an intentionally unsafe or incomplete answer. Cleanup or rollback: remove raw transcripts after redaction, delete temporary files that contain private data, and keep only the anonymized dataset and its change history.

Assessment Exercises

• You have 500 support conversations and time to label only 40. Propose a sampling strategy that covers frequency, severity, and known failures.
• A model-based grader prefers longer answers even when they contain unsupported claims. What calibration rows and deterministic checks would you add?
• Your regression score improves, but high-risk policy questions regress. How should the dataset report be changed so this cannot hide inside the average?
• Design a rubric for a retrieval-augmented answer that must cite evidence and refuse when the evidence is insufficient.
• Explain when you would move a development row into the stable regression set.

Summary

Building an evaluation dataset is the act of defining quality in executable form. The dataset should contain representative tasks, hard cases, rubrics, context, metadata, and split assignments. Real incidents give relevance, synthetic cases fill gaps, and stable regression rows prevent old failures from returning. The most important habit is to write the rubric before comparing systems, then preserve the row history so every score can be explained.