CloudWatch Metrics, Logs, Alarms, and Traces

CloudWatch is the operational telemetry layer used throughout AWS Cloud Engineering work: it stores numerical measurements as metrics, indexed event records as logs, threshold logic as alarms, and request paths as traces. The practical outcome is not simply a dashboard. A well-built CloudWatch design lets you answer four questions quickly: is the workload healthy, what changed, which dependency is involved, and what action should happen next?

In this course section on operations and delivery, CloudWatch connects infrastructure choices to release confidence. EC2, Lambda, API Gateway, Application Load Balancers, ECS, RDS, and many other services publish telemetry into CloudWatch automatically, but useful operations still require intentional names, dimensions, retention, filters, alarm thresholds, and trace sampling. This lesson focuses on those mechanics and the design decisions behind them.

How CloudWatch Represents Workload Behavior

A CloudWatch metric is a time-ordered set of data points identified by a namespace, a metric name, and zero or more dimensions. The namespace groups related producers, such as AWS/Lambda or AWS/ApplicationELB. Dimensions are key-value labels, such as FunctionName or LoadBalancer, that form a specific metric identity. Errors for one Lambda function and Errors for another are separate time series because their dimensions differ.

Each metric data point has a timestamp, unit, and value or statistic set. CloudWatch aggregates data into periods, then alarms evaluate statistics such as Average, Sum, Minimum, Maximum, or percentiles. Resolution matters: standard metrics usually work at one-minute granularity, while custom high-resolution metrics can be stored at finer intervals. Higher resolution can reduce detection time, but it increases cost and can expose more noise.

CloudWatch Logs stores text or JSON events inside log streams, which belong to log groups. A common pattern is one log group per application component and one log stream per instance, container task, or execution environment. Retention is set at the log group level. Without an explicit retention policy, logs can remain much longer than intended, which affects both cost and data governance.

Alarms are state machines. They evaluate one metric, math expression, anomaly detection band, or Metrics Insights query over a configured number of periods. The main states are OK, ALARM, and INSUFFICIENT_DATA. Alarms do not continuously inspect every raw event; they evaluate aggregated data points on a schedule. That distinction explains many surprises when a metric appears in a graph but an alarm has not changed state yet.

Traces show a single request as segments and subsegments across services. In AWS, CloudWatch ServiceLens and the X-Ray data model are commonly used together: instrumented code sends trace segments, managed integrations add service nodes, and the resulting service map helps connect latency or errors to a specific hop. Metrics say something is wrong, logs provide local detail, and traces show where a request spent time.

Configuration Anatomy

A useful CloudWatch configuration names the producer, the signal, the aggregation, and the response. For metrics, this means choosing namespace, metric name, dimensions, unit, period, and statistic. For logs, it means choosing log group name, JSON field names, retention days, metric filters, and access controls. For alarms, it means choosing threshold, comparison operator, evaluation periods, datapoints to alarm, treatment of missing data, and actions. For traces, it means choosing sampling rules, segment names, annotations for indexed search, metadata for deeper debugging, and propagation of trace headers across service calls.

CloudWatch is regional. Metrics, log groups, alarms, and traces are stored in the Region where they are emitted unless you intentionally centralize or cross-account aggregate them. Names should make ownership and environment clear, for example /aws/lambda/payments-prod-authorizer rather than lambda-logs. Dimensions should identify useful slices without exploding cardinality. A request ID is excellent in a log event, but usually a poor metric dimension because it creates a new time series for almost every request.

Example 1: Custom Metric for Checkout Failures

Suppose a checkout worker detects a failed payment authorization. The worker can publish a custom metric with the service and environment as dimensions. The expected behavior is deterministic: the command returns no metric value directly, but a later query for the same namespace, name, and dimensions should show a data point with value 1.

set -euo pipefail

aws cloudwatch put-metric-data \
  --namespace "Course/Checkout" \
  --metric-data '[{"MetricName":"PaymentAuthorizationFailures","Dimensions":[{"Name":"Service","Value":"checkout"},{"Name":"Environment","Value":"dev"}],"Unit":"Count","Value":1}]'

aws cloudwatch get-metric-statistics \
  --namespace "Course/Checkout" \
  --metric-name "PaymentAuthorizationFailures" \
  --dimensions Name=Service,Value=checkout Name=Environment,Value=dev \
  --start-time "2026-09-06T00:00:00Z" \
  --end-time "2026-09-07T00:00:00Z" \
  --period 300 \
  --statistics Sum \
  --output json

The important design choice is the dimension set. Service and Environment make the signal actionable without creating unbounded time series. Adding CustomerId as a metric dimension would make dashboards expensive and difficult to reason about. Put customer or request identifiers in structured logs instead.

Example 2: Logs Insights for Slow Requests

Structured JSON logs make CloudWatch Logs Insights far more useful than plain strings. In this example, application logs include route, status, and duration_ms. The query filters for slow successful requests, groups by route, and returns the slowest route averages first.

fields @timestamp, route, status, duration_ms
| filter status = 200 and duration_ms > 1000
| stats count(*) as slow_requests, avg(duration_ms) as avg_ms, max(duration_ms) as max_ms by route
| sort avg_ms desc
| limit 10

If the log group contains events such as {"route":"/checkout","status":200,"duration_ms":1450} and {"route":"/cart","status":200,"duration_ms":300}, only /checkout contributes to the result because the cart event does not pass the duration filter. This is a progressive step beyond metrics: metrics can show a latency alarm, while logs explain which route or operation is causing it.

Example 3: Alarm for Lambda Error Rate

Raw error counts can be misleading when traffic changes. Metric math can calculate an error rate from Lambda Errors and Invocations. The alarm below enters ALARM when more than 5 percent of invocations fail for two out of three evaluation periods.

{
  "AlarmName": "course-checkout-dev-lambda-error-rate",
  "ComparisonOperator": "GreaterThanThreshold",
  "EvaluationPeriods": 3,
  "DatapointsToAlarm": 2,
  "Threshold": 5,
  "TreatMissingData": "notBreaching",
  "Metrics": [
    {
      "Id": "errors",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Lambda",
          "MetricName": "Errors",
          "Dimensions": [{ "Name": "FunctionName", "Value": "checkout-dev-handler" }]
        },
        "Period": 60,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "invocations",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Lambda",
          "MetricName": "Invocations",
          "Dimensions": [{ "Name": "FunctionName", "Value": "checkout-dev-handler" }]
        },
        "Period": 60,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "error_rate",
      "Expression": "IF(invocations>0,100*errors/invocations,0)",
      "Label": "Error rate percent",
      "ReturnData": true
    }
  ]
}

Expected behavior is clear: if the function has 100 invocations and 8 errors in a one-minute period, the expression produces 8. If that happens in at least two of the last three periods, the alarm moves to ALARM. If there are no invocations, the expression returns 0, preventing divide-by-zero behavior from creating noise.

Example 4: Trace a Request Across Services

For distributed requests, traces complement both metrics and logs. A checkout API might call a pricing service, a payment provider, and a DynamoDB table. The trace should use annotations for searchable fields such as route and tenant_tier, while keeping high-cardinality details as metadata.

def annotate_checkout_trace(segment, route, tenant_tier, cart_id):
    segment.put_annotation("route", route)
    segment.put_annotation("tenant_tier", tenant_tier)
    segment.put_metadata("cart_id", cart_id, "checkout")
    return {"trace_annotations": {"route": route, "tenant_tier": tenant_tier}}

result = annotate_checkout_trace(segment=type("Segment", (), {"put_annotation": lambda self, k, v: None, "put_metadata": lambda self, k, v, n: None})(), route="/checkout", tenant_tier="standard", cart_id="cart-123")
print(result)

The deterministic output is {'trace_annotations': {'route': '/checkout', 'tenant_tier': 'standard'}}. The design lesson is that annotations are indexed for trace filtering, so they should be bounded and useful. Metadata can carry diagnostic details, but it is not the right place for secrets, payment data, or unbounded payloads.

Design Choices and Trade-offs

Start with user-visible symptoms, then map them to signals. Availability often needs alarms on error rate, dependency failures, and synthetic checks. Performance needs latency percentiles, queue depth, saturation, and trace segments. Cost control needs log retention, metric cardinality discipline, and sampling choices. Security needs careful log redaction, least-privilege read access, and separation between application operators and audit log administrators.

Alarm sensitivity is a trade-off between detection speed and fatigue. A one-minute, one-datapoint alarm catches issues quickly but can page on a short retry storm. A five-minute alarm with multiple datapoints is calmer but detects incidents later. TreatMissingData should match the workload: breaching may be right for a heartbeat metric where silence is bad, while notBreaching often fits sparse error metrics.

Dashboards are helpful for diagnosis but should not be the primary incident detector. Alarms should have owners, actions, and runbooks. Logs should be queryable by stable fields. Traces should be sampled enough to diagnose common paths and high-value failures without collecting every request unnecessarily.

Failure Modes and Troubleshooting

Symptom: an alarm never fires even though errors appear in logs. The cause is often a mismatch between log events and metric identity, or an alarm watching the wrong dimensions. Diagnose by graphing the exact namespace, metric name, dimensions, statistic, and period used by the alarm. Correct by publishing the metric with the same dimensions or updating the alarm to match the real producer.

Symptom: an alarm stays in INSUFFICIENT_DATA. Common causes include sparse metrics, delayed ingestion, an incorrect Region, or a period shorter than the metric resolution. Check the metric graph in the same Region as the alarm and verify recent data points. Correct by adjusting period and missing-data treatment, or by emitting a heartbeat metric when absence of data is meaningful.

Symptom: Logs Insights queries are slow and expensive. The cause is usually broad time ranges, too many log groups, or unstructured messages that require scanning large volumes. Diagnose by narrowing the time window and checking whether fields are auto-discovered from JSON. Correct by logging structured events, selecting fewer log groups, and applying filters before aggregation.

Symptom: traces show missing downstream calls. The cause may be missing instrumentation, lost trace headers, unsupported client libraries, or sampling that drops the request. Diagnose by checking whether the incoming request has a trace header and whether each service creates a segment or subsegment. Correct by enabling supported instrumentation, propagating headers through queues or HTTP calls, and reviewing sampling rules.

Security, Performance, and Reliability Implications

CloudWatch data often contains operationally sensitive information. Logs can accidentally include tokens, customer data, or full request bodies. Use structured logging with explicit allowlists, redact before emission, encrypt log groups where required, and restrict logs:GetLogEvents, logs:StartQuery, and trace-read permissions to the people and roles that need them.

From a performance perspective, synchronous custom metric calls or trace submissions should not become a request bottleneck. Prefer asynchronous emission where the runtime supports it, batch custom metrics when practical, and set timeouts for telemetry clients. From a reliability perspective, telemetry should degrade gracefully: an application should not fail checkout solely because a metric could not be submitted.

Hands-on Lab: Build a Small CloudWatch Signal

Prerequisites: an AWS account, AWS CLI credentials for a sandbox environment, permission to use CloudWatch metrics, alarms, and logs, and a Region selected with AWS_REGION or your CLI profile.

  1. Create or choose a sandbox log group named /course/cloudwatch/dev and set a short retention period appropriate for temporary work.
  2. Publish one custom metric named LabFailures in namespace Course/Lab with dimensions Service=orders and Environment=dev.
  3. Create an alarm named course-lab-orders-failures that evaluates the Sum of LabFailures over one-minute periods and alarms when the value is greater than zero.
  4. Write two JSON log events: one normal order event and one failed order event with fields order_id, status, and duration_ms.
  5. Run a Logs Insights query that filters for status = "failed" and returns the failed order_id.

Verification: confirm that get-metric-statistics returns the custom metric data point, the alarm eventually reaches ALARM after the breaching data point is evaluated, and the Logs Insights query returns only the failed order. If you enabled trace instrumentation, verify that the service map shows the lab service and any instrumented downstream call.

Cleanup: delete the alarm, delete or shorten retention on the lab log group, and stop publishing the custom metric. Custom metrics age out after publication stops, but alarms and log groups remain billable resources until removed or retained according to policy.

Assessment Exercises

  1. A Lambda function has low traffic overnight. Should its error-count alarm use TreatMissingData=breaching, notBreaching, or another design? Justify the answer based on what missing data means for that function.
  2. Design metric dimensions for an API latency metric. Explain which labels belong in dimensions and which belong in logs because of cardinality.
  3. An operator sees elevated p95 latency but no increase in average latency. What should they inspect in traces and logs before changing capacity?
  4. Write a Logs Insights query that finds the top three failing routes from JSON logs containing route and status.
  5. Explain why an alarm on raw error count can be less useful than an alarm on error rate during a traffic spike.

Summary

CloudWatch works by turning service behavior into metrics, log events, alarm state, and request traces. The engineering value comes from choosing stable dimensions, structured log fields, meaningful thresholds, and trace annotations that match real failure modes. In AWS cloud operations, this is the feedback system that lets teams release, diagnose, and recover with evidence rather than guesswork.