Deploying Agents to Production

Deploying an agent means running it as a reliable service that real users and real systems depend on. An agent that works in a notebook can still fail in production: models time out, tools return errors, users send hostile input, and a loop can spend money for hours. This lesson shows the practical steps that turn a demo into something you can safely operate.

Overview: why agents are harder to deploy than normal apps

A normal web service is mostly deterministic: the same input gives the same output. An agent is not. It calls a language model (LLM), the model decides which tool to use, and the tool may change the outside world. Three properties make production harder:

• Non-determinism. The same request can take 2 steps one day and 9 steps the next.
• Side effects. Tools can send emails, write to databases or issue refunds. A mistake is not just a wrong answer; it is a wrong action.
• Variable cost and latency. Every step is a paid network call. A runaway loop is a billing problem as well as a bug.

So production deployment is mostly about adding boundaries around the agent: limits on how long and how much it can run, controls on what it may do, visibility into what it did, and a safe way to roll back. Think of the LLM as an untrusted, creative component inside a strict, boring shell. The shell is what you deploy.

The production checklist

Key concepts

Configuration and secrets

Follow the twelve-factor idea: the same code runs in development, staging and production, and only the environment changes. Model name, limits and keys come from environment variables. If a required value is missing, the program should crash at startup with a clear message, not fail halfway through a customer request. Secrets must never appear in logs, error messages or Git history.

Bounded execution

Every agent loop needs a maximum number of steps, a total time limit and a spending limit. Every network call needs a timeout. When a limit is hit, the agent should stop and return a clear failure, not keep trying.

Retries with backoff

Transient errors (rate limits, brief network failures) are normal. Retry them a small number of times, waiting longer each time (exponential backoff), ideally with a little random jitter so many clients do not retry at the same instant. Do not retry errors that will never succeed, such as an invalid request. Retry only tools that are idempotent (safe to run twice); a refund tool needs an idempotency key or it must not be retried blindly.

Human approval and least privilege

Give the agent only the tools it needs, with the narrowest permissions. Classify tools as read-only or risky. Risky tools (payments, deletes, outbound messages) should pause for a human decision, or be limited by hard rules such as a maximum amount.

Prompt-injection defence

Text from web pages, emails, documents and tool results can contain instructions aimed at your agent. Treat all such content as data, never as commands. The most reliable defence is not a clever prompt but structure: least-privilege tools, approval gates, and validation of tool arguments in ordinary code.

Examples

Example 1: fail-fast configuration

This loads settings from the environment, refuses to start if required values are missing, and never prints the API key. The demo passes a dictionary so it runs anywhere.

import os
from dataclasses import dataclass


@dataclass(frozen=True)
class Settings:
    model: str
    max_steps: int
    timeout_s: float
    api_key: str

    @classmethod
    def from_env(cls, env=os.environ):
        missing = [k for k in ("AGENT_MODEL", "LLM_API_KEY") if not env.get(k)]
        if missing:
            raise RuntimeError("Missing required settings: " + ", ".join(missing))
        return cls(
            model=env["AGENT_MODEL"],
            max_steps=int(env.get("AGENT_MAX_STEPS", "8")),
            timeout_s=float(env.get("AGENT_TIMEOUT_S", "30")),
            api_key=env["LLM_API_KEY"],
        )

    def safe_repr(self):
        return (f"Settings(model={self.model!r}, max_steps={self.max_steps}, "
                f"timeout_s={self.timeout_s}, api_key='***')")


demo_env = {"AGENT_MODEL": "your-model-name", "LLM_API_KEY": "placeholder-key", "AGENT_MAX_STEPS": "5"}
print(Settings.from_env(demo_env).safe_repr())
try:
    Settings.from_env({})
except RuntimeError as exc:
    print("startup failed:", exc)

Output:

Settings(model='your-model-name', max_steps=5, timeout_s=30.0, api_key='***')
startup failed: Missing required settings: AGENT_MODEL, LLM_API_KEY

The dataclass is frozen, so settings cannot change while the program runs. safe_repr hides the key, so logging the settings is safe. In real deployments the platform (Kubernetes, a cloud service, Docker) injects the variables from its secret store.

Example 2: a bounded, retrying agent loop with structured logs

Here call_llm() is a stub that first fails once (simulating a rate limit), then asks for a tool, then gives a final answer. To use a real provider, replace the body of call_llm() with your SDK call and convert its response into the same small dictionaries. Network errors that are safe to retry should be raised as TransientError.

import json
import time


class TransientError(Exception):
    pass


_calls = {"n": 0}


def call_llm(messages):
    """Stub. Replace with a real SDK call that returns the same shapes."""
    _calls["n"] += 1
    if _calls["n"] == 1:
        raise TransientError("rate limited")
    tool_results = [m for m in messages if m["role"] == "tool"]
    if not tool_results:
        return {"type": "tool_call", "name": "get_order_status", "args": {"order_id": "A100"}}
    return {"type": "final", "text": "Order A100 has shipped."}


def get_order_status(order_id):
    return {"order_id": order_id, "status": "shipped"}


TOOLS = {"get_order_status": get_order_status}


def log(event, **fields):
    print(json.dumps({"event": event, **fields}, sort_keys=True))


def call_with_retry(messages, attempts=3, base_delay=0.01):
    for attempt in range(1, attempts + 1):
        try:
            return call_llm(messages)
        except TransientError as exc:
            log("llm_retry", attempt=attempt, error=str(exc))
            if attempt == attempts:
                raise
            time.sleep(base_delay * 2 ** (attempt - 1))


def run_agent(question, max_steps=5):
    messages = [{"role": "user", "content": question}]
    for step in range(1, max_steps + 1):
        reply = call_with_retry(messages)
        if reply["type"] == "final":
            log("done", steps=step)
            return reply["text"]
        tool = TOOLS.get(reply["name"])
        if tool is None:
            raise RuntimeError("unknown tool: " + reply["name"])
        result = tool(**reply["args"])
        log("tool_call", step=step, name=reply["name"])
        messages.append({"role": "tool", "content": json.dumps(result)})
    raise RuntimeError("step limit reached")


print(run_agent("Where is order A100?"))

Output:

{"attempt": 1, "error": "rate limited", "event": "llm_retry"}
{"event": "tool_call", "name": "get_order_status", "step": 1}
{"event": "done", "steps": 2}
Order A100 has shipped.

Walk through it: the first LLM call fails, the retry wrapper logs it, waits briefly and tries again. The second call returns a tool request, which the loop runs. The third call sees the tool result and returns a final answer. If the model kept asking for tools forever, the for loop would end after max_steps and raise an error. Logs are one JSON object per line, which log systems can search and aggregate easily.

Example 3: a human-approval gate and audit trail

Risky tools go through an approver function. In production the approver could post to a review queue or chat channel and wait for a person. Here it is a simple rule so the example runs offline. Every decision is recorded.

import json

RISKY = {"issue_refund"}
audit = []


def issue_refund(order_id, amount):
    return f"refunded {amount} for {order_id}"


def lookup(order_id):
    return f"order {order_id}: delivered"


TOOLS = {"issue_refund": issue_refund, "lookup": lookup}


def execute(name, args, approver):
    if name not in TOOLS:
        audit.append({"tool": name, "outcome": "unknown"})
        return "error: unknown tool"
    if name in RISKY and not approver(name, args):
        audit.append({"tool": name, "args": args, "outcome": "denied"})
        return "error: human approval denied"
    result = TOOLS[name](**args)
    audit.append({"tool": name, "args": args, "outcome": "ok"})
    return result


def deny_all(name, args):
    return False


def allow_small(name, args):
    return args.get("amount", 0) <= 50


print(execute("lookup", {"order_id": "A100"}, deny_all))
print(execute("issue_refund", {"order_id": "A100", "amount": 500}, allow_small))
print(execute("issue_refund", {"order_id": "A100", "amount": 20}, allow_small))
for entry in audit:
    print(json.dumps(entry, sort_keys=True))

Output:

order A100: delivered
error: human approval denied
refunded 20 for A100
{"args": {"order_id": "A100"}, "outcome": "ok", "tool": "lookup"}
{"args": {"amount": 500, "order_id": "A100"}, "outcome": "denied", "tool": "issue_refund"}
{"args": {"amount": 20, "order_id": "A100"}, "outcome": "ok", "tool": "issue_refund"}

Read-only lookup runs freely, even with deny_all. The 500 refund is blocked, and the 20 refund passes. Notice that the gate lives in ordinary code, outside the model. Even if a prompt-injected document convinces the model to request a huge refund, the code still refuses. The audit list answers the question “what did the agent do, and who allowed it?”

Example 4: packaging in a container

A container makes the runtime identical everywhere. This Dockerfile installs dependencies first (so Docker can cache that layer) and runs as a non-root user.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN useradd --create-home agent
USER agent
EXPOSE 8000
CMD ["python", "server.py"]

Build and run it, passing secrets at run time from an env file that is excluded from Git:

docker build -t support-agent:1.0 .
docker run --rm -p 8000:8000 --env-file ./agent.env support-agent:1.0
curl -s http://localhost:8000/healthz

The last command assumes your server exposes a simple /healthz endpoint that returns success when the process is alive. Orchestrators use such endpoints to restart broken instances.

Under the hood: what happens to one production request

1. The request arrives and gets a unique run ID, attached to every log line.
2. Input is validated (size, format, user identity and permissions).
3. The loop starts with a step counter, a deadline and a budget.
4. Each LLM call has a timeout and retry policy; token usage is recorded.
5. Each tool call is checked against an allow-list, its arguments are validated, and risky ones wait for approval.
6. The loop ends with a final answer or a clean failure; the result and audit record are stored.
7. Metrics (latency, steps, errors, cost) are updated so dashboards and alerts can see trends.

Monitoring and release

Track: success rate, steps per run, latency, tool error rate, tokens and cost per run, and how often humans deny approvals. Set alerts on sudden changes. Before each release, run a fixed set of test tasks (an evaluation set) and compare results with the current version. Roll out to a small share of traffic first, and keep the previous image ready for rollback. Model providers update and retire models over time, so pin the model name in configuration and check your provider’s documentation for current model names and deprecation notices.

Common Mistakes

Mistake 1: secrets in code and an unbounded loop

API_KEY = "sk-real-looking-key"   # committed to Git

def run(question):
    messages = [question]
    while True:                    # no step limit, no timeout
        reply = call_llm(messages)
        messages.append(reply)

Why it is wrong: anyone with repository access can read the key, and a model that never gives a final answer will loop forever and run up costs. Fix: read the key from the environment (Example 1) and use a for loop with max_steps (Example 2).

Mistake 2: retrying everything, including side effects

for attempt in range(5):
    try:
        issue_refund(order_id, amount)
        break
    except Exception:
        continue

Why it is wrong: if the refund succeeded but the response was lost, the retry refunds the customer again. Catching every Exception also hides real bugs. Fix: retry only specific transient errors, and give risky tools an idempotency key so repeated calls have one effect.

Mistake 3: trusting content the agent reads

Letting a web page or email decide which tool runs, with no approval gate, means any author of that page can control your agent. Keep the gate in code, as in Example 3, and treat retrieved text as untrusted data.

Best Practices

• Load all configuration from environment variables and fail at startup if required values are missing.
• Store secrets in a secret manager; rotate them; never log them.
• Set limits on steps, wall-clock time, tokens and spend per run and per day.
• Use timeouts on every network call, and retry only transient, idempotent operations with backoff and jitter.
• Give each tool the least privilege it needs; use separate read-only and write credentials.
• Require human approval (or hard rule limits) for payments, deletions, external messages and anything irreversible.
• Validate tool arguments in code with an allow-list of tools; never run model output as a shell command or SQL directly.
• Log one structured line per step with a run ID, and keep an audit trail of every side-effecting action.
• Run as a non-root user in a container and expose a health endpoint.
• Keep an evaluation set, run it before every release, and roll out gradually with a quick rollback path.
• Have a kill switch (a flag that disables risky tools or the whole agent) that you can flip without redeploying.
• Be careful with personal data in logs; redact it and follow your retention rules.

Practice Exercises

1. Add a budget. Extend run_agent from Example 2 so each LLM reply carries a made-up cost number, and stop with an error when the running total passes a limit given as a parameter. Hint: keep a spent variable and check it inside the loop.
2. Persistent audit log. Change Example 3 so each audit entry is appended as one JSON line to a file, together with a timestamp from time.time(). Make sure no secret values are written.
3. Health endpoint. Using only the standard library (http.server), write a tiny server that answers /healthz with the text ok and any other path with status 404. Then write the Docker commands to run it.

Summary

• Production agents are an LLM inside a strict shell of limits, permissions and monitoring.
• Configuration comes from the environment; secrets stay out of code, and the app fails fast when settings are missing.
• Bound every run: step limit, timeouts, time and spending budgets.
• Retry only transient, idempotent failures with exponential backoff.
• Enforce safety in code: least-privilege tools, argument validation, human approval for risky actions, and treating external text as untrusted data.
• Log structured events and keep an audit trail so every action can be explained.
• Package in a container, evaluate before release, roll out gradually, and keep a rollback and kill switch ready.