Agent Loops and State Machines

An agent loop lets a generative AI system repeatedly inspect a situation, choose the next move, run a permitted action, and update state until it reaches a terminal outcome. A state machine makes that loop explicit: every turn has a named state, allowed transitions, stored data, budgets, and termination rules. The practical outcome is an agent that can do multi-step work without becoming an unbounded chat transcript that happens to call tools.

In this course, agent loops sit after tool calling because a loop is what decides whether another tool call is needed. The model may help classify the next action or draft the final response, but the software owns the loop, the state record, the tool permissions, and the stop conditions. That division is what turns a language model from a conversational component into a controlled workflow participant.

Purpose and Outcome

The purpose of an agent loop is to decompose uncertain work into bounded steps. A support assistant might read a user request, retrieve an order, inspect shipping exceptions, offer a policy-compliant answer, and stop. A coding assistant might inspect files, decide which command to run, read the result, patch a file, test, and stop. In both cases, the loop is not merely repetition. It is a controller that keeps track of what has been observed, what has been tried, what remains allowed, and what final condition has been satisfied.

A state machine gives that controller a vocabulary. Common states include observe, plan or decide, act, verify, done, and failed. Transitions describe which state may follow which. The state payload stores data such as the user goal, retrieved evidence, selected action, tool outputs, retry counters, token budget, elapsed time, and final answer. Terminal states are especially important: done means the goal was satisfied, while failed means the loop stopped intentionally with a known failure category.

Internal Mechanism

A minimal loop has four phases. In the observation phase, the controller gathers the current input: the user message, previous state, tool results, and any external facts loaded for this step. In the decision phase, it chooses an action. That choice might be a deterministic rule, a model-generated structured object, or a hybrid where rules handle policy and the model handles semantic interpretation. In the action phase, the system executes only the selected permitted operation. In the update phase, the controller records the result and moves to the next state.

The key internal detail is that the model should not be the loop itself. The model can propose {"action":"search","query":"..."}, but application code validates that object, checks whether search is legal in the current state, decrements a budget, executes the tool, stores the observation, and decides whether the next state is terminal. This separation prevents hidden prompt text from becoming the only description of the workflow.

State may be ephemeral for a single request or durable across a longer job. Ephemeral state is simpler and often enough for synchronous tasks. Durable state is useful when a workflow waits on a user, schedules background work, survives process restarts, or must be auditable. Durable state should include a schema version, because future code may need to resume a job created by older code. For deterministic replay, store tool inputs and outputs, not just a summary written by the model.

API Anatomy

A practical agent loop usually has these interfaces: a state type, a transition function, an action schema, a tool registry, and a runner. The state type names the current state and carries the payload. The transition function maps state + observation to the next state. The action schema is the narrow contract the model or rules may produce. The tool registry maps action names to executable functions with permission metadata. The runner enforces budgets, catches classified failures, and emits the final response.

Budgets are part of the API, not an afterthought. Useful budgets include maximum loop steps, maximum tool calls, maximum retries per tool, maximum tokens per model call, maximum wall-clock time, and sometimes maximum cost. A loop without a budget can repeat because the model keeps requesting more information, a tool keeps returning ambiguous data, or the verification phase never accepts the answer. The budget should produce a clear failure result rather than silently truncating the workflow.

Example 1: A Deterministic State Machine

The first example uses no model at all. That is intentional: it shows the controller shape without probabilistic behavior. The transition function allows only known states and produces either DONE or FAILED.

from dataclasses import dataclass, replace
from enum import Enum, auto

class AgentState(Enum):
    OBSERVE = auto()
    DECIDE = auto()
    ACT = auto()
    DONE = auto()
    FAILED = auto()

@dataclass(frozen=True)
class AgentFrame:
    state: AgentState
    user_goal: str
    observation: str = ""
    decision: str = ""
    answer: str = ""
    steps: int = 0
    max_steps: int = 4

def transition(frame: AgentFrame) -> AgentFrame:
    if frame.steps >= frame.max_steps:
        return replace(frame, state=AgentState.FAILED, answer="step budget exhausted")
    if frame.state is AgentState.OBSERVE:
        return replace(frame, state=AgentState.DECIDE, observation="goal: " + frame.user_goal, steps=frame.steps + 1)
    if frame.state is AgentState.DECIDE:
        decision = "finish" if "capital" in frame.observation.lower() else "ask_tool"
        return replace(frame, state=AgentState.ACT, decision=decision, steps=frame.steps + 1)
    if frame.state is AgentState.ACT and frame.decision == "finish":
        return replace(frame, state=AgentState.DONE, answer="Paris", steps=frame.steps + 1)
    return replace(frame, state=AgentState.FAILED, answer="unsupported decision: " + frame.decision)

frame = AgentFrame(state=AgentState.OBSERVE, user_goal="What is the capital of France?")
while frame.state not in {AgentState.DONE, AgentState.FAILED}:
    frame = transition(frame)
print(frame.state.name, frame.answer, frame.steps)

The expected output is DONE Paris 3. The example is small, but it demonstrates three production ideas: terminal states are explicit, each transition increments the step counter, and an unsupported decision becomes a classified failure instead of another improvised action.

Example 2: A Tool-Using Agent Loop

The second example adds a tool registry and a scratchpad. The scratchpad is not a free-form prompt; it is stored state containing tool observations. The decision function first checks whether evidence already exists. If not, it selects a permitted tool and argument.

from dataclasses import dataclass, replace
from typing import Callable

@dataclass(frozen=True)
class LoopState:
    goal: str
    scratchpad: tuple[str, ...] = ()
    final: str | None = None
    calls_left: int = 2

TOOLS: dict[str, Callable[[str], str]] = {
    "lookup_order": lambda order_id: "order A-100 is delayed by weather",
}

def choose_action(state: LoopState) -> tuple[str, str]:
    if state.scratchpad:
        return ("final", "Tell the customer: " + state.scratchpad[-1])
    if "A-100" in state.goal and state.calls_left > 0:
        return ("lookup_order", "A-100")
    return ("final", "I need an order id before I can check status.")

def run_agent(goal: str) -> LoopState:
    state = LoopState(goal=goal)
    while state.final is None and state.calls_left >= 0:
        action, argument = choose_action(state)
        if action == "final":
            return replace(state, final=argument)
        if action not in TOOLS:
            return replace(state, final="No permitted tool named " + action)
        result = TOOLS[action](argument)
        state = replace(state, scratchpad=state.scratchpad + (result,), calls_left=state.calls_left - 1)
    return replace(state, final="Unable to finish within the tool budget.")

print(run_agent("Where is order A-100?").final)

The expected output is Tell the customer: order A-100 is delayed by weather. In a model-backed version, choose_action might call a model that returns structured JSON. The surrounding runner should stay similar: validate the action name, validate the argument, execute a registered tool, append the observation, and continue only while the budget allows.

Example 3: Retry State and Recovery

The third example isolates retry state. Retries should belong to a tool call or transition, not to the whole conversation. Otherwise, one flaky dependency can consume the entire agent budget and hide which operation actually failed.

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class RetryState:
    attempts: int = 0
    max_attempts: int = 3
    last_error: str = ""
    result: str = ""

def flaky_tool(attempt: int) -> str:
    if attempt < 2:
        raise TimeoutError("inventory service timed out")
    return "12 units available"

def call_with_retry() -> RetryState:
    state = RetryState()
    while state.attempts < state.max_attempts:
        attempt = state.attempts + 1
        try:
            return replace(state, attempts=attempt, result=flaky_tool(attempt), last_error="")
        except TimeoutError as exc:
            state = replace(state, attempts=attempt, last_error=str(exc))
    return state

state = call_with_retry()
print(state.result, state.attempts, state.last_error or "ok")

The expected output is 12 units available 2 ok. The state records that the first call timed out and the second succeeded. A real implementation would usually add backoff, idempotency keys for write actions, and a retry policy that distinguishes timeouts from validation errors.

Design Choices and Trade-offs

The first design choice is deterministic versus model-selected transitions. Deterministic transitions are easier to test and should handle safety gates, permissions, budgets, and terminal conditions. Model-selected transitions are useful when the input requires semantic judgment, such as deciding whether a customer is asking for a refund, a shipping update, or a policy exception. A robust design often uses the model to fill a narrow action schema while deterministic code decides whether that action is legal.

The second choice is graph shape. A simple loop is enough for many agents: observe, decide, act, update, repeat. More complex workflows use a directed graph, where retrieval, approval, tool execution, verification, and human handoff are separate nodes. Graphs make branching clearer, but they can become difficult to reason about if every edge contains special cases. Keep transition rules close to the state they affect.

The third choice is how much history to include in each model call. Passing the whole transcript is convenient but expensive and can reintroduce stale or irrelevant observations. Passing a compact state summary is cheaper, but summaries can omit details needed for correctness. For important workflows, store full observations in durable state and pass the model a carefully selected working context.

Failure Modes and Troubleshooting

A common symptom is an agent that keeps calling the same tool. The cause is usually that the decision phase never sees the previous tool result, or the verification rule always rejects the answer. Diagnose it by printing or tracing state after every transition: current state, selected action, argument, calls left, and last observation. Correct it by appending tool output to state before the next decision and by adding a terminal rule for sufficient evidence.

Another symptom is a final answer that claims an action happened when no tool ran. The cause is often allowing the model to write natural language instead of a validated action object. Diagnose it by checking whether every external claim in the response corresponds to a stored tool result. Correct it by requiring structured actions, refusing unknown action names, and generating the final answer from state rather than from unsupported conversation text.

A third symptom is duplicate purchases, duplicate emails, or repeated ticket updates. The cause is retrying non-idempotent actions after a timeout. The tool may have succeeded, but the caller did not receive the response. Diagnose it by comparing action logs for repeated arguments and timestamps. Correct it with idempotency keys, a read-after-write check, and a retry policy that treats write timeouts differently from read timeouts.

Security, Performance, and Reliability

Agent loops increase impact because they can chain actions. Least privilege should be expressed in the tool registry: an agent that checks order status does not need a refund tool unless the current state and user role allow refunds. Treat tool outputs as untrusted input, because retrieved web pages, tickets, or documents may contain instructions aimed at the model. The controller should decide which fields are evidence and which fields are merely content.

Performance depends on step count, model calls, and tool latency. A five-step loop with three model calls can feel slow even when each component is healthy. Use early exits for simple cases, parallelize independent reads where the workflow permits it, and cache stable observations. Reliability improves when every terminal failure carries a category such as budget_exhausted, tool_timeout, invalid_action, or approval_required.

Hands-on Lab

Prerequisites: Python 3.10 or newer, a terminal, and no external services. Create a file named agent_loop_lab.py and paste the code from Example 2. Run python agent_loop_lab.py. Verify that the output says Tell the customer: order A-100 is delayed by weather.

  1. Change the goal to Where is my order? and rerun the file. The expected final message is I need an order id before I can check status.
  2. Change calls_left in LoopState from 2 to 0 and use the original goal. The expected behavior is that the agent does not call the tool and asks for an order id, because the decision function requires remaining calls.
  3. Add a second tool named lookup_policy that returns weather delays are not refundable until day 5. Extend choose_action so that after the order lookup it calls the policy tool before finalizing. Verify that the scratchpad contains two observations before the final response.
  4. Cleanup is simply deleting agent_loop_lab.py. If you added package dependencies while experimenting, remove them from your virtual environment or recreate the environment.

Assessment Exercises

  • Design a state machine for an agent that schedules a meeting. Which states require deterministic transitions, and where could a model safely classify intent?
  • In Example 2, what bug would appear if tool output were appended to a local variable but not returned in the next LoopState?
  • Choose budgets for a refund-processing agent: steps, tool calls, retries, and human approvals. Explain the risk each budget controls.
  • Rewrite Example 3 so that write actions are never automatically retried after a timeout. What state would you need to store?
  • Given a trace with ten repeated lookup_order calls and no final answer, list the first three fields you would inspect and why.

Summary

Agent loops coordinate multi-step model behavior by making every observe, decide, act, and update cycle explicit. State machines provide the named states, allowed transitions, budgets, and terminal outcomes that keep the loop inspectable. In generative AI systems, the strongest pattern is to let the model propose narrow structured decisions while deterministic code validates actions, runs tools, records observations, and decides when the workflow is done or failed.