Systems Manager, Patching, and Fleet Operations

AWS Systems Manager gives cloud engineers a control plane for operating fleets of EC2 instances, edge servers, and hybrid machines without signing in to each host. In this lesson, the outcome is concrete: you should be able to enroll managed nodes, understand how Systems Manager knows their state, patch them in controlled batches, run commands safely, and prove what happened afterward.

In the Compute and Scaling section of this AWS Cloud course, Systems Manager matters because Auto Scaling groups, launch templates, and immutable images do not remove every host operation. Long-running fleets still need emergency commands, software inventory, patch compliance, session access, and coordinated maintenance. The point is not to click Patch now; it is to design fleet operations that are scoped, auditable, reversible, and boring during an incident.

How Systems Manager Works

Systems Manager is a regional AWS service that coordinates actions through the SSM Agent. The agent runs on a managed node, registers with the Systems Manager endpoint, receives work, executes plugins locally, and returns status. On EC2, the instance profile normally grants the node permission to call Systems Manager, EC2 messages, and Systems Manager messages APIs. For hybrid machines, activation credentials register the machine as a managed instance.

The important internal split is between control plane intent and host-local execution. You define intent in AWS: a command document, a patch baseline, an association, a maintenance window, or an automation runbook. The agent performs the operating-system work: shell commands, package-manager calls, PowerShell scripts, inventory collection, file downloads, and reboot detection. If the agent is stopped, the instance can be perfectly healthy for serving traffic but invisible to fleet operations.

Several Systems Manager resources appear repeatedly. A Document describes steps to run. AWS-managed documents such as AWS-RunShellScript and AWS-RunPatchBaseline cover common operations, while custom documents encode team-specific procedures. An Association applies a document on a schedule or as a desired-state rule. State Manager owns those associations. Patch Manager evaluates missing and installed patches against patch baselines. Maintenance Windows bind tasks to approved time periods. Inventory records package, application, network, and instance metadata. Session Manager provides shell access without opening SSH or RDP inbound ports.

Configuration Anatomy

A managed node needs three things before higher-level features work. First, the agent must be installed and running. Many AWS AMIs include it, but custom images and on-premises machines must be checked. Second, network paths must reach Systems Manager endpoints. Public internet access through NAT works, but private subnets commonly use VPC interface endpoints for ssm, ec2messages, and ssmmessages; CloudWatch Logs and S3 endpoints are also common when commands write logs or fetch artifacts. Third, IAM must allow only the needed actions. AWS managed policies are useful for learning, but production roles should separate node registration, operator commands, automation administration, and read-only audit access.

Patching adds another layer. A Patch Baseline defines which patches are approved, rejected, delayed, or selected by classification and severity. A Patch Group is a tag value that maps nodes to a baseline. For example, production Linux web servers can use one baseline while developer Windows hosts use another. A Maintenance Window then chooses when to scan or install and how many targets can run at the same time using concurrency and error thresholds.

Run Command uses three selection fields that deserve careful review: the document name, the targets, and the parameters. The document decides what the agent can execute. Targets decide which nodes receive it, often by tag such as Role=web. Parameters pass the actual shell command or patch operation. Output can be returned inline for small results, written to S3, streamed to CloudWatch Logs, or queried later through command invocation status.

Example 1: Discover Managed Nodes

The first example performs read-only discovery. It shows whether Systems Manager can see nodes tagged as web servers and returns the identifiers that later commands will target.

set -euo pipefail

aws ssm describe-instance-information \
  --filters Key=tag:Role,Values=web \
  --query 'InstanceInformationList[].{Id:InstanceId,Platform:PlatformType,Ping:PingStatus,Agent:AgentVersion}' \
  --output table

Expected behavior is a table with one row per managed node. PingStatus should be Online before command execution or patching. If no rows appear, do not assume there are no web servers; it may mean the tag is absent, the agent is not registered, or the current AWS Region is wrong.

Example 2: Scan Patch Compliance

The second example scans for missing patches without installing them. This is the right next step because it builds evidence before changing a host. AWS-RunPatchBaseline invokes the local package manager, compares results with the node’s patch baseline, and reports compliance.

set -euo pipefail

COMMAND_ID=$(aws ssm send-command \
  --document-name AWS-RunPatchBaseline \
  --targets Key=tag:PatchGroup,Values=linux-web \
  --parameters Operation=Scan \
  --max-concurrency 25% \
  --max-errors 1 \
  --query Command.CommandId \
  --output text)

aws ssm list-command-invocations \
  --command-id "$COMMAND_ID" \
  --details \
  --query 'CommandInvocations[].{Instance:InstanceId,Status:Status,Response:StatusDetails}' \
  --output table

The deterministic expectation is the command life cycle: invocations move from Pending or InProgress to Success, Failed, TimedOut, or Cancelled. Compliance results are then available in Systems Manager Compliance. Because this is a scan, packages should not be installed and hosts should not reboot.

Example 3: Patch in a Maintenance Window

The third example shows the operational pattern for production: define a maintenance window, register tagged targets, and attach an install task with conservative batching. The schedule shown is a placeholder cron expression; teams should align it with their own change calendar.

set -euo pipefail

WINDOW_ID=$(aws ssm create-maintenance-window \
  --name course-linux-web-patching \
  --schedule 'cron(0 4 ? * SUN *)' \
  --duration 2 \
  --cutoff 1 \
  --allow-unassociated-targets \
  --query WindowId \
  --output text)

TARGET_ID=$(aws ssm register-target-with-maintenance-window \
  --window-id "$WINDOW_ID" \
  --resource-type INSTANCE \
  --targets Key=tag:PatchGroup,Values=linux-web \
  --owner-information course \
  --query WindowTargetId \
  --output text)

aws ssm register-task-with-maintenance-window \
  --window-id "$WINDOW_ID" \
  --targets Key=WindowTargetIds,Values="$TARGET_ID" \
  --task-type RUN_COMMAND \
  --task-arn AWS-RunPatchBaseline \
  --max-concurrency 10% \
  --max-errors 1 \
  --priority 1 \
  --task-invocation-parameters '{"RunCommand":{"Parameters":{"Operation":["Install"],"RebootOption":["RebootIfNeeded"]}}}'

When the window opens, Systems Manager starts the task only for the registered targets. With 10% concurrency, a fleet of 50 nodes patches about five at a time. With max-errors set to 1, the window stops scheduling new invocations after the threshold is crossed, limiting blast radius.

Design Choices and Trade-Offs

The first design choice is mutable patching versus replacing instances from a patched image. Image replacement is cleaner for Auto Scaling workloads because new nodes start from known artifacts. Patch Manager is still useful for stateful servers, mixed operating systems, hybrid nodes, emergency zero-day response, and compliance reporting. Many teams use both: monthly golden AMIs for routine updates and Systems Manager patching for exceptions or long-lived infrastructure.

The second choice is targeting by tags versus explicit instance IDs. Tags scale better and follow replacement instances, but a wrong tag can affect many hosts. Explicit IDs are safer for one-off repair but brittle for fleets. Mature operations use tag governance, read-only dry checks, and narrow patch groups such as linux-web-prod-a rather than broad labels like prod.

The third choice is direct commands versus Automation runbooks. Run Command is simple for short host-level tasks. Automation is better for multi-step workflows such as taking an AMI, draining an instance from a load balancer, patching, health checking, and restoring traffic. Automation steps have clearer inputs, approvals, retries, and rollback points, but they require more up-front modeling.

Failure Modes and Troubleshooting

Symptom: nodes are missing from describe-instance-information. Cause: the SSM Agent is stopped, the instance profile lacks permissions, the Region is wrong, or private networking cannot reach the service endpoints. Diagnostic steps: check the EC2 instance profile, verify the agent service locally if you can access the host, inspect VPC endpoints or NAT routing, and query by instance ID instead of tag. Correction: attach a least-privilege Systems Manager role, start or update the agent, add the required endpoints, and standardize tags in the launch template.

Symptom: patch commands fail with package-manager errors. Cause: OS repositories are unreachable, package locks are held by another process, disk is full, or the baseline rejects required dependencies. Diagnostic steps: inspect command plugin output, CloudWatch Logs, and the host’s package-manager logs. Check free disk and whether another update service is running. Correction: restore repository access, clear stale locks only after confirming no active update, expand disk or clean caches, and adjust baseline rules through review.

Symptom: the command succeeds but the application is unhealthy afterward. Cause: a required reboot occurred, a service failed to restart, configuration drift existed before patching, or too many nodes left service at once. Diagnostic steps: compare command timestamps with load balancer target health, application logs, and system boot time. Correction: lower concurrency, add prechecks and postchecks, drain nodes before patching, and move the process into an Automation runbook with rollback.

Security, Reliability, and Performance

Session Manager can reduce network exposure by removing inbound administrative ports, but it also centralizes powerful access. Require IAM conditions, logging to CloudWatch Logs or S3, encryption with KMS where appropriate, and separation between users who can start sessions and users who can alter logging configuration. For Run Command, restrict who can use shell documents against production tags.

Reliability comes from batching, error thresholds, health checks, and rollback. Patching every node at once is rarely justified. For performance, remember that inventory and compliance queries are control-plane operations, while patch installation consumes host CPU, disk, network, and repository capacity. Schedule scans and installs so they do not compete with peak workload traffic or backup windows.

Hands-On Lab: Controlled Patch Scan

Prerequisites: an AWS account, AWS CLI configured for a test Region, one test EC2 instance with SSM Agent online, an instance profile that permits Systems Manager managed-instance operations, and a tag PatchGroup=linux-web. Use a non-production host.

  1. Confirm the node is visible with aws ssm describe-instance-information --filters Key=tag:PatchGroup,Values=linux-web.
  2. Run the scan command from Example 2 and save the returned command ID.
  3. Query list-command-invocations until the status is terminal.
  4. Open Systems Manager Compliance and filter by the instance ID to view missing patch counts.
  5. Change nothing else, then rerun the scan to confirm repeatability.

Verification: the command invocation reaches Success, the instance remains reachable, no reboot occurs, and compliance data appears for the managed node. Cleanup: remove any temporary tags, delete test maintenance windows created during practice, and detach broad trial policies if you replaced them with narrower production roles.

Assessment Exercises

  1. You have 200 web instances behind a load balancer. Design a patch window with concurrency, error threshold, and health verification. Explain what stops the rollout.
  2. A security team asks for proof that a critical package is installed everywhere. Which Systems Manager data sources and commands would you use, and what evidence would be weak?
  3. A Run Command target selector uses Environment=prod. Identify two risks and propose a safer targeting scheme.
  4. An instance serves traffic but never receives commands. Walk through the diagnostic path from IAM to agent to network endpoint.
  5. Choose between Patch Manager and immutable AMI replacement for a stateful database helper node. Defend the choice and name the rollback path.

Summary

Systems Manager turns individual host administration into regional fleet operations. Its mechanics are agent registration, document-driven execution, tag-based targeting, patch baselines, maintenance windows, and auditable command status. Good AWS engineers use those mechanics deliberately: discover before changing, patch in small batches, log administrative access, diagnose agent and network failures directly, and connect every fleet action to health verification and rollback.