Scoring Quality and Groundedness
Quality scoring tells you whether a model response is useful for the task. Groundedness scoring tells you whether the response is supported by the evidence the system was allowed to use. In generative AI engineering, you need both. A response can be fluent and well structured while inventing facts, and a response can quote evidence accurately while failing to answer the user’s actual question.
The outcome of this lesson is a practical scoring design: define an answerable task, separate quality from groundedness, verify citations against retrieved evidence, use model-assisted grading only where deterministic checks are insufficient, and calibrate scores against human review. This belongs in the Evaluation and Safety section because scoring is the feedback loop that reveals whether retrieval, prompting, tool use, and response formatting are working together.
What Quality and Groundedness Measure
Quality is task-relative. For a customer-support answer, quality may include correct resolution steps, tone, completeness, and escalation behavior. For a research assistant, it may include coverage, uncertainty handling, source selection, and concise synthesis. Quality cannot be measured by a single universal number without losing important detail, so production systems usually use a rubric with named dimensions.
Groundedness is evidence-relative. It asks whether each factual claim in the answer is entailed by the supplied context, contradicted by it, or not found. This differs from truth. A claim may be true in the world but ungrounded if the retrieved documents did not support it. That distinction matters because retrieval-augmented generation, tool answers, and compliance workflows often require the system to prove where its information came from.
Scoring Internals
A scoring pipeline usually has four stages. First, it normalizes the task record: user question, model answer, retrieved passages, tool results, citations, policy labels, and metadata such as prompt version. Second, it applies deterministic checks: schema validity, required fields, citation identifiers, answer length, refusal format, and whether cited source IDs exist. Third, it applies semantic checks: claim-to-evidence matching, rubric grading, pairwise comparison, or model-as-judge classification. Fourth, it aggregates results into metrics such as grounded claim rate, unsupported citation rate, rubric mean, pass rate, and task completion rate.
The unit of scoring should be explicit. Answer-level scoring is simple but hides mixed behavior. Citation-level scoring checks whether each citation points to relevant evidence. Claim-level scoring is more precise because one paragraph may contain supported and unsupported claims. Task-level scoring asks whether the user outcome was achieved, such as booking completed, correct SQL produced, or safe refusal returned.
Model-assisted grading is useful when the criterion requires language understanding, but it must be constrained. Give the grader the question, answer, evidence, rubric, and allowed labels. Ask for structured output rather than prose. Keep the grader independent from the generator when possible. Track grader version, prompt version, and calibration examples because judge behavior can drift when models or instructions change.
Rubric and API Anatomy
A usable rubric names dimensions, score ranges, evidence requirements, and failure conditions. A common pattern is a one-to-five quality score plus separate binary or three-way groundedness labels. For groundedness, labels such as supported, unsupported, and contradicted are easier to audit than vague numeric scores. For quality, dimensions such as answer_correctness, completeness, instruction_following, and clarity are more actionable than a single satisfaction score.
The scoring record should include the raw answer, evidence identifiers, parsed citations, individual dimension scores, aggregate decision, and explanation fields that are short enough for review. Do not let the explanation replace the score. The explanation helps debugging, while the score drives dashboards, gates, and regression tests.
Example 1: Deterministic Rubric Checks
The first layer catches issues that do not require a language model. This example checks required answer sections and validates that every citation marker references a known source. It produces deterministic output, which makes it suitable for unit tests and release gates.
import re
KNOWN_SOURCES = {"doc-1", "doc-2"}
REQUIRED_SECTIONS = ("answer", "evidence")
def deterministic_score(answer: str) -> dict[str, object]:
lower = answer.lower()
missing_sections = [name for name in REQUIRED_SECTIONS if f"{name}:" not in lower]
citations = re.findall(r"\[(doc-\d+)\]", answer)
unknown = sorted({source for source in citations if source not in KNOWN_SOURCES})
return {
"passes": not missing_sections and not unknown and bool(citations),
"missing_sections": missing_sections,
"unknown_citations": unknown,
"citation_count": len(citations),
}
sample = "Answer: Use the billing export. Evidence: The export includes invoice totals [doc-1]."
print(deterministic_score(sample))The expected result is a pass with one citation and no missing sections. If the answer cited [doc-9], the same check would fail before any expensive judge call. This is not a complete quality score, but it is a cheap guardrail that prevents malformed answers from polluting semantic evaluation.
Example 2: Citation Grounding
The next layer checks whether cited evidence actually contains the core claim. Production systems may use entailment models or claim extraction, but a small deterministic version is useful for understanding the mechanism. Here the answer is split into cited claims, and each claim is checked against the cited passage using required terms.
EVIDENCE = {
"doc-1": "The billing export includes invoice totals, customer IDs, and payment status.",
"doc-2": "Refunds are processed from the adjustments screen after manager approval.",
}
CLAIMS = [
{"text": "The billing export includes invoice totals", "source": "doc-1", "terms": {"billing", "export", "invoice", "totals"}},
{"text": "Refunds require manager approval", "source": "doc-2", "terms": {"refunds", "manager", "approval"}},
]
def groundedness_score(claims: list[dict[str, object]], evidence: dict[str, str]) -> dict[str, object]:
results = []
for claim in claims:
passage = evidence.get(str(claim["source"]), "").lower()
terms = {str(term).lower() for term in claim["terms"]}
missing = sorted(term for term in terms if term not in passage)
results.append({"claim": claim["text"], "source": claim["source"], "label": "supported" if not missing else "unsupported", "missing_terms": missing})
supported = sum(1 for result in results if result["label"] == "supported")
return {"supported_claims": supported, "total_claims": len(results), "claim_results": results}
print(groundedness_score(CLAIMS, EVIDENCE)["supported_claims"])The expected output is 2. The mechanism is intentionally simple: it treats support as term coverage. Real systems need stronger semantic matching because evidence can support a claim with synonyms or numeric transformations. However, this example shows the design boundary: a groundedness score should connect a specific claim to specific evidence, not merely ask whether the whole answer sounds plausible.
Example 3: Calibrated Model-Assisted Grading
A judge model is most valuable for nuanced quality dimensions. The engineering challenge is not calling a model; it is making the judging task stable enough to compare over time. The following stand-in function behaves deterministically, but it mirrors the structured output you should request from a real judge.
def judge_quality(question: str, answer: str, grounded_rate: float) -> dict[str, object]:
score = 1
reasons = []
if len(answer.split()) >= 12:
score += 1
else:
reasons.append("answer is too brief")
if "next step" in answer.lower() or "because" in answer.lower():
score += 1
else:
reasons.append("answer lacks explanation or action")
if grounded_rate >= 0.8:
score += 2
else:
reasons.append("groundedness is below threshold")
return {"quality_score": min(score, 5), "passes": score >= 4, "reasons": reasons}
answer = "Use the billing export because it includes invoice totals, then verify payment status as the next step."
print(judge_quality("How do I review invoice totals?", answer, 1.0))The expected score is five with passes set to true. With a real judge, keep several human-labeled calibration examples for each score level. Run the judge against those examples whenever you change the prompt, rubric, model, or evidence formatting. If the judge disagrees with trusted labels, fix the grader before using it to evaluate the generator.
Design Choices and Trade-offs
Strict citation checking reduces unsupported claims but can penalize useful synthesis when evidence is split across passages. Claim-level scoring is more accurate than answer-level scoring but costs more because it requires claim extraction and more review surface. A single aggregate score is easy for dashboards but poor for debugging; dimension scores are better for engineering decisions. Human review is expensive and slower, but it is essential for calibration, disputed cases, and high-impact domains.
Thresholds should map to decisions. For example, a groundedness rate below 0.8 might block release in a regression suite, while a live answer with one unsupported minor claim might be routed to a fallback response. Do not tune thresholds only to improve a benchmark. Tune them against the cost of false passes and false failures in the actual product.
Failure Modes and Troubleshooting
One common symptom is a high quality score with user complaints about hallucination. The cause is often a rubric that rewards fluency and completeness without a separate groundedness gate. Diagnose by sampling failed answers, marking each factual claim, and checking whether the cited passages support it. Correct the issue by splitting quality and groundedness into separate dimensions and preventing unsupported high-confidence claims from passing.
Another symptom is a sudden drop in groundedness after retrieval changes. The cause may be chunking that separates definitions from qualifiers, or ranking that retrieves related but non-answering passages. Diagnose by logging retrieved source IDs, passage text, citation IDs, and claim labels for a stable evaluation set. Correct by adjusting chunk boundaries, retrieval filters, reranking, or prompt instructions that require citing only passages used in the answer.
A third symptom is judge scores changing even though generated answers look similar. The cause may be grader prompt drift, model changes, temperature, or hidden formatting changes in the evaluation harness. Diagnose by rerunning calibration examples and comparing structured judge outputs. Correct by pinning the rubric text, recording grader metadata, using deterministic settings where available, and reviewing examples whose labels changed.
Security, Performance, and Reliability
Scoring systems often handle sensitive prompts, documents, and generated answers. Store only what you need for evaluation, redact secrets before sending records to a judge model, and avoid exposing private evidence in explanations shown to end users. If human reviewers are involved, assign queues by permission and remove data that is not needed for the scoring decision.
Performance matters because scoring can double or triple model traffic. Use deterministic checks first, then send only eligible cases to expensive semantic graders. Batch offline evaluation, cache scores for unchanged task records, and keep live scoring lightweight unless the score affects immediate safety. Reliability depends on treating evaluation data as versioned assets: stable datasets, rubric versions, scorer versions, and repeatable reports.
Hands-on Lab: Build a Mini Scoring Harness
Prerequisites: Python 3, a terminal, and no external API keys. Create a small script with three records. Each record should contain a question, an answer, evidence passages, and expected labels. Step one: implement deterministic citation validation. Step two: list claims and source IDs. Step three: compute a groundedness rate. Step four: apply a quality rubric that cannot pass when groundedness is below your threshold. Step five: print a report with per-record status and aggregate pass rate.
records = [
{"id": "case-1", "grounded_rate": 1.0, "quality_score": 5},
{"id": "case-2", "grounded_rate": 0.5, "quality_score": 4},
{"id": "case-3", "grounded_rate": 1.0, "quality_score": 3},
]
def final_decision(record: dict[str, object]) -> str:
if float(record["grounded_rate"]) < 0.8:
return "fail_groundedness"
if int(record["quality_score"]) < 4:
return "fail_quality"
return "pass"
for record in records:
print(record["id"], final_decision(record))
passes = sum(1 for record in records if final_decision(record) == "pass")
print(f"pass_rate={passes / len(records):.2f}")Verification: the first case should pass, the second should fail groundedness even though its quality score is high, the third should fail quality even though it is grounded. Cleanup is simple: delete the temporary script and any local reports. If you adapt the lab to real model calls, also remove temporary files containing prompts, documents, or answers that should not be retained.
Assessment Exercises
- Given an answer with three factual claims and two citations, design a claim-level scoring table that distinguishes supported, unsupported, and contradicted claims.
- Your judge model gives consistently higher scores than human reviewers for verbose answers. Identify two rubric changes and one calibration test that would reduce that bias.
- A retrieval update improves answer completeness but lowers citation precision. Explain how you would decide whether to ship it.
- Design a release gate that uses deterministic checks, groundedness rate, quality score, and a small human-reviewed sample without collapsing them into one opaque number.
- For a medical or legal assistant, explain which scoring failures should trigger refusal, human escalation, or product rollback.
Summary
Scoring quality and groundedness is the discipline of turning subjective-looking model behavior into auditable measurements. Quality asks whether the answer solves the task. Groundedness asks whether its factual claims are supported by the supplied evidence. The strongest systems combine deterministic validation, claim-to-evidence checks, calibrated judge models, human review, and task outcome metrics. Keep dimensions separate, version your rubrics and scorers, troubleshoot with concrete claims and sources, and make every threshold correspond to a product decision.
