Service Metrics, Logs, Traces, and SLOs

Service metrics, logs, traces, and service level objectives are the operating instruments for an ML prediction service. They answer different questions. Metrics show aggregate health over time, logs preserve selected facts about individual events, traces connect work across components, and SLOs turn measurements into an explicit reliability promise. The outcome of this lesson is practical: you should be able to instrument a model endpoint so an operator can tell whether users are affected, where the delay or failure is happening, which model and data contract were involved, and whether the service is consuming its error budget too quickly.

In the MLOps lifecycle, this lesson sits after deployment because a trained model is not finished when it reaches production. The online service can be healthy while predictions degrade, or the model can be accurate while the API is too slow to be useful. Good telemetry joins service behavior with model context without exposing sensitive payloads.

Telemetry Roles

A metric is a numeric time series identified by a name and labels, such as prediction_requests_total{route="/predict",status="200",model_version="risk-v12"}. Counters only increase, gauges move up and down, and histograms bucket observations such as latency. Metrics are cheap to query for dashboards and alerts, but they lose per-request detail.

A log is a timestamped event. In ML services, logs should be structured records with fields for request identity, model version, feature schema, outcome class, latency, and failure category. They should not contain raw features, secrets, or free-form exception dumps that include customer data. Logs are strongest when investigating a known request or explaining a small group of unusual outcomes.

A trace is a tree of spans sharing a trace identifier. One root span might represent POST /predict; child spans represent feature lookup, validation, inference, policy checks, and audit write. Traces explain causality and latency composition. They are especially useful when a prediction path crosses an API gateway, feature store, model server, and event sink.

An SLO is a target over a measured service level indicator. For example, 99.9 percent of prediction requests over 30 days return a non-5xx response within 300 ms. The SLI is the measured ratio; the SLO is the chosen target; the error budget is the allowed amount of bad service. SLOs help teams decide when to ship changes, slow releases, or focus on reliability work.

Internal Mechanism

Instrumentation starts at request entry. The service creates or accepts a correlation identifier, starts a root span, increments an in-flight gauge, and records the request start time. Validation then tags the request with low-cardinality labels: route, method, tenant tier if appropriate, model version, and schema version. High-cardinality values such as user ID, email, raw prompt, account number, or full feature vector do not belong in metric labels because they explode time series cardinality and create privacy risk.

When the request finishes, the service records a counter for total requests, a counter for failures by category, and a histogram observation for latency. Logs are emitted at important state transitions: validation rejected, feature lookup timed out, inference completed, fallback used, audit write failed. The trace exports spans with durations and attributes. A collector or agent receives telemetry, batches it, and forwards it to storage. Dashboards query metrics; log search indexes selected fields; tracing storage keeps sampled request graphs.

SLO evaluation usually runs from metrics, not logs. For a request-based availability SLO, the numerator is good events and the denominator is total eligible events. For latency, histogram buckets can calculate the fraction under threshold. Alerts often use burn rate: how quickly the service consumes its error budget compared with the budgeted rate. A fast-burn page catches major incidents; a slower ticket catches chronic degradation.

Configuration Anatomy

A useful telemetry design names four things before any tool choice. First, define the signal names: request count, error count, latency histogram, prediction distribution, feature missing rate, drift statistic, and queue depth. Second, define labels with bounded values: route, status class, model version, schema version, dependency, and failure category. Third, define log fields: timestamp, severity, event name, request ID, trace ID, model version, input schema, output class, and sanitized reason. Fourth, define SLO windows, eligibility rules, thresholds, and the owner who responds.

The most common design mistake is mixing debugging detail into every telemetry channel. Metrics should stay small and aggregate. Logs should be structured and searchable. Traces should be sampled enough to explain representative behavior. SLOs should describe user-visible outcomes, not every internal component alarm.

Example 1: RED Metrics for Prediction Traffic

The RED pattern records rate, errors, and duration. For an ML endpoint, add model version and schema version when the number of active values is controlled. The example below calculates a tiny deterministic snapshot. Real systems export these observations continuously to a metrics backend.

from collections import Counter

requests = [
    {"route": "/predict", "status": 200, "latency_ms": 42},
    {"route": "/predict", "status": 200, "latency_ms": 55},
    {"route": "/predict", "status": 503, "latency_ms": 18},
]

count = Counter(item["status"] for item in requests)
error_rate = sum(v for k, v in count.items() if k >= 500) / len(requests)
latencies = sorted(item["latency_ms"] for item in requests)
p95 = latencies[int(0.95 * (len(latencies) - 1))]
print(f"requests_total={len(requests)} errors={error_rate:.2%} p95_ms={p95}")

The output is requests_total=3 errors=33.33% p95_ms=42. The p95 calculation is intentionally simple for three observations; production histogram queries approximate percentiles from buckets. The important behavior is that one failed request changes the aggregate error rate without storing the request payload.

Example 2: Structured Prediction Logs

Logs fill the detail gap left by metrics. This event records what happened, which deployed artifact made the decision, and how long the request took. It excludes input features and customer identifiers.

import json

log_event = {
    "level": "info",
    "event": "prediction_completed",
    "request_id": "req-7f3",
    "model_version": "credit-risk-2026-08-14",
    "feature_schema": "application-v4",
    "latency_ms": 51,
    "decision": "approved",
}

print(json.dumps(log_event, sort_keys=True))

The output is one JSON object with stable keys. In a log backend, an operator can search event=prediction_completed and model_version=credit-risk-2026-08-14, then compare latency or decision mix across versions. Because the record is structured, it can be filtered reliably without parsing prose.

Example 3: Trace Spans Across the Prediction Path

When latency rises, a single duration metric does not say which component slowed down. A trace breaks one request into spans.

from dataclasses import dataclass

@dataclass(frozen=True)
class Span:
    name: str
    start_ms: int
    end_ms: int
    parent: str | None = None

trace = [
    Span("http POST /predict", 0, 82),
    Span("load features", 6, 28, "http POST /predict"),
    Span("model inference", 30, 63, "http POST /predict"),
    Span("write audit event", 65, 80, "http POST /predict"),
]

for span in trace:
    indent = "  " if span.parent else ""
    print(f"{indent}{span.name}: {span.end_ms - span.start_ms} ms")

The expected output lists the root request at 82 ms, then child spans for feature loading at 22 ms, inference at 33 ms, and audit writing at 15 ms. If only the feature span grows during an incident, the likely owner is the feature store or its network path, not the model artifact.

Example 4: SLO Error Budget Math

An SLO converts telemetry into an operational decision. A 99.9 percent availability target allows 0.1 percent bad requests in the measurement window.

total_requests = 1_000_000
allowed_bad = int(total_requests * 0.001)  # 99.9% availability target
observed_5xx = 640
remaining_budget = allowed_bad - observed_5xx
burn_rate = observed_5xx / allowed_bad
print(f"allowed_bad={allowed_bad} remaining={remaining_budget} burn_rate={burn_rate:.2f}x")

The output is allowed_bad=1000 remaining=360 burn_rate=0.64x. This means 640 bad requests have consumed 64 percent of the monthly budget. A release policy might allow normal deployment below 50 percent, require extra review above 75 percent, and freeze risky changes after the budget is exhausted.

Design Choices and Trade-offs

Choose labels carefully. Adding model_version helps compare canary and stable releases, but adding request_id to a metric creates a new time series for every request. Sample traces based on traffic and risk. Head-based sampling is cheap but may miss rare failures; tail-based sampling can retain slow or failed traces but requires buffering decisions. Log at boundaries instead of every function. Excessive logging raises cost, hides important events, and increases the chance of sensitive data exposure.

SLOs should be user-centered. A CPU alert is useful for diagnosis, but users care about predictions being available, timely, and valid. Separate service SLOs from model-quality monitors. Accuracy, calibration, drift, and fairness signals often need delayed labels or batch analysis, so they should not be forced into the same minute-by-minute SLO as API availability.

Failure Modes and Troubleshooting

Symptom: dashboards become slow and storage cost jumps. Cause: a high-cardinality label such as user ID, prompt hash, or request ID was added to a metric. Diagnosis: inspect top metric names by active series and group by label cardinality. Correction: remove the label, keep the value only in sampled logs or traces when policy allows it, and add a lint rule for metric labels.

Symptom: users report slow predictions, but service error rate is normal. Cause: latency SLO was missing or measured only at the load balancer while downstream feature lookup slowed. Diagnosis: compare edge latency with application spans and dependency metrics. Correction: add an end-to-end latency SLI and trace spans around feature retrieval, inference, and post-processing.

Symptom: a model canary looks successful, then complaints rise after full rollout. Cause: service telemetry was healthy but model outcome distribution shifted for a segment. Diagnosis: compare prediction distribution, missing-feature rate, and delayed outcome metrics by model version and schema version. Correction: roll back the model version, restore the previous feature contract, and add segment-level model monitors with privacy-approved aggregation.

Security, Performance, and Reliability

Telemetry is production data. Apply retention limits, access controls, and redaction before export. Use allowlisted log fields, not best-effort string scrubbing after a record is built. Keep trace attributes bounded and avoid raw examples. Performance overhead should be measured: synchronous log writes or unbounded telemetry queues can make an incident worse. Use batching, backpressure, and drop policies that prefer losing debug detail over blocking predictions. Reliability improves when alerts are tied to SLO burn rate and runbooks, because responders see both urgency and likely starting points.

Hands-on Lab: Instrument a Minimal Predictor

Prerequisites: Python 3, a shell, and permission to run local scripts. No external service is required. Steps: create a temporary Python file, paste the four examples in order, and run it. Add a fourth request with status 200 and latency 310 ms. Recalculate the error rate and decide whether this request is availability-good and latency-good for a 300 ms latency SLO. Add a new structured log field named fallback_used with a boolean value. Add a span named fallback lookup only when fallback is true. Finally, compute an SLO with 50,000 requests and 120 bad events for a 99.5 percent target.

Verification: the script should print deterministic metric, log, trace, and budget lines. The added 310 ms request should not increase availability errors, but it should count as bad for a 300 ms latency SLI. The 99.5 percent target allows 250 bad events, so 120 observed bad events leaves 130. Cleanup: delete the temporary script and any local output files. If you used a real telemetry sandbox, remove test metric names and dashboards to avoid confusing future reviews.

Assessment Exercises

  1. A team wants to label every metric with customer account ID so support can debug faster. Explain the operational and privacy consequences, then propose a safer design.
  2. Your 5xx error rate is flat, but the p95 prediction latency doubled after a feature store migration. Which traces and metrics would you inspect first, and why?
  3. Define one availability SLO and one latency SLO for a fraud scoring API. State the numerator, denominator, window, and alert condition for each.
  4. A canary model has lower API latency but a higher missing-feature rate. Decide whether to continue rollout, roll back, or gather more evidence, and justify the decision.

Summary

Metrics, logs, traces, and SLOs are complementary mechanisms, not interchangeable dashboards. Metrics show aggregate health, logs explain selected events, traces reveal cross-service work, and SLOs convert measurements into reliability decisions. For ML services, the useful design joins these signals to model version, schema version, feature health, and prediction outcomes while keeping sensitive data out of telemetry.