Guardrails and Red Teaming

Test prompt injection, sensitive-data disclosure, harmful requests, tool misuse, and encoding tricks while monitoring false refusals.

This lesson is part of the Evaluation and Safety section. Measure failures continuously and layer controls according to real risk. It develops the topic from first principles through implementation, failure modes, verification, and production decisions so you can explain not only what to do, but why.

Overview: How Guardrails and Red Teaming Works

A generative AI application is a probabilistic model inside a deterministic software system. Reliability comes from controlling context, tools, schemas, permissions, evaluation, and monitoring rather than assuming a fluent answer is a correct answer.

For this topic, the central objective is to test prompt injection, sensitive-data disclosure, harmful requests, tool misuse, and encoding tricks while monitoring false refusals. Start by defining the observable outcome and the trust boundaries. Then identify inputs, transformations, state, outputs, and failure paths. This prevents a demonstration that works once from being mistaken for a system that behaves correctly under changing data, concurrency, malformed input, and dependency failures.

Key vocabulary

Term Meaning in this course
token a unit of model input or output rather than necessarily a whole word
context the instructions and evidence available for the current generation
embedding a vector representation used to compare semantic similarity
evaluation a repeatable measurement of behavior on representative tasks

Architecture and Decision Process

  1. Write the user or system outcome in a form that can be tested.
  2. List trusted and untrusted inputs, including data produced by dependencies.
  3. Choose the smallest abstraction that expresses the behavior clearly.
  4. Validate before expensive work or irreversible state changes.
  5. Return a bounded, documented result and record enough telemetry to diagnose failure.
  6. Test normal behavior, edge cases, permission failures, timeouts, and recovery.

Each step is deliberately observable. If a result is wrong, you should be able to determine whether the input contract, transformation, dependency, policy, or presentation caused it. Hidden global state and broad exception handling make that diagnosis harder and should be minimized.

Implementation Pattern

The first example gives the feature an explicit input contract and keeps validation close to the boundary:

from dataclasses import dataclass

@dataclass(frozen=True)
class GenerationRequest:
    task: str
    evidence: tuple[str, ...]
    max_output_tokens: int = 321

def build_guardrails_and_red_teaming_request(question: str) -> GenerationRequest:
    if not question.strip():
        raise ValueError("question is required")
    return GenerationRequest(task=question, evidence=())

Read the types as part of the design, not decoration. They communicate required data to humans and tools, but runtime checks still belong at boundaries. Keep domain decisions independent from transport or provider details so the behavior can be tested without a network connection.

Verification Pattern

The second example makes one important property executable:

def evaluate_guardrails_and_red_teaming(answer: str, required_terms: set[str]) -> dict[str, object]:
    normalized = answer.lower()
    missing = sorted(term for term in required_terms if term.lower() not in normalized)
    return {"passed": not missing, "missing": missing, "lesson": 21}

result = evaluate_guardrails_and_red_teaming("Evidence and citations are required.", {"evidence", "citations"})
print(result["passed"])

A syntax check proves only that Python can parse the source. A useful test also checks the documented contract, representative data, and a meaningful failure case. Production confidence comes from layers: fast unit tests, boundary integration tests, a small end-to-end suite, and monitored real-world outcomes.

What Happens Step by Step

  1. The caller supplies data under a documented schema and identity context.
  2. The boundary rejects missing, malformed, oversized, or unauthorized input.
  3. Application logic applies the lesson’s core operation without leaking infrastructure concerns.
  4. Dependencies are called with explicit timeouts, bounded retries, and least-privilege credentials.
  5. The result is validated again before it becomes a response, stored record, or automated action.
  6. Metrics and structured events record latency, success, failure category, and version without secrets.

Common Mistakes

  • Starting with a fashionable library or model before defining the problem and acceptance criteria.
  • Validating only the user interface while trusting direct API calls, stored data, or dependency output.
  • Using a broad catch-all exception that turns programming defects into plausible but incorrect results.
  • Testing with a tiny happy-path sample that does not represent production users or data.
  • Logging credentials, personal data, prompts, documents, or raw payloads without a retention policy.

Another subtle failure is coupling evaluation to implementation. If a metric rewards the shortcut your current system already takes, an apparent improvement may not improve the actual user outcome. Keep a stable external definition of success and add new regression cases whenever a real failure is discovered.

Best Practices

  • Prefer explicit schemas, versioned configuration, and deterministic preprocessing.
  • Separate policy and domain logic from frameworks, vendors, databases, and user-interface code.
  • Use least privilege, deny by default, and require human approval for high-impact actions.
  • Set resource budgets for time, memory, requests, retries, and output size.
  • Measure quality, latency, cost, and safety on representative workloads before and after release.
  • Design rollback and degraded behavior before the first production incident.

Practice Exercises

  • Draw the complete data flow for Guardrails and Red Teaming, marking every trust boundary and persistent state change.
  • Add one invalid-input test, one dependency-failure test, and one authorization test to the example.
  • Define a production metric, an alert threshold, and the operator action that should follow the alert.
  • Explain which part you would replace when requirements change and which contract should remain stable.

Production Checklist

  • Inputs, outputs, owners, and data retention are documented.
  • Secrets are stored outside source code and scoped to the minimum capability.
  • Tests cover expected, adversarial, and degraded behavior.
  • Dashboards distinguish user errors, internal failures, dependency failures, and policy refusals.
  • A rollback or safe-disable mechanism has been exercised.

Summary

  • Guardrails and Red Teaming should be designed around a measurable contract rather than a demonstration.
  • The key focus is to test prompt injection, sensitive-data disclosure, harmful requests, tool misuse, and encoding tricks while monitoring false refusals.
  • Explicit boundaries, representative evaluation, least privilege, observability, and rollback turn the concept into dependable production engineering.