CI-CD, Deployment Strategies, and Immutable Artifacts

CI/CD on AWS turns a source change into a verified production change through repeatable stages. In this lesson the outcome is narrow and practical: build one immutable artifact, promote that exact artifact through environments, and choose a deployment strategy that limits user impact when the new version is wrong. The important habit is separating building from releasing. A release should point infrastructure at a known artifact, not rebuild from a moving branch.

What The Pipeline Is Responsible For

A typical AWS delivery path has four responsibilities. Continuous integration checks a proposed change by compiling, testing, scanning, and packaging it. Artifact storage records the build result in a place such as Amazon ECR for container images, Amazon S3 for ZIP files or deployment bundles, and AWS CodeArtifact for packages. Continuous delivery prepares a deployment and waits for approval or automated policy. Continuous deployment completes the promotion without a manual gate when tests and health checks pass.

In an AWS cloud engineering workflow this sits between development and operations. IAM decides which stage can build, read, or deploy. Networking and load balancers decide how traffic moves. CloudWatch alarms, logs, and deployment events decide whether the release continues or rolls back. The pipeline is therefore part of the workload architecture, not a separate admin script.

Internal Mechanics

CodePipeline models a release as an execution flowing through ordered stages. A source action produces a revision identifier, such as a Git commit. A build action, commonly CodeBuild, runs inside an ephemeral build environment using a service role. The build writes output artifacts to an artifact bucket and can publish an image to ECR. Later actions consume those artifacts by name. This is why artifact identity matters: the deploy stage should receive a digest, versioned object key, or build output from the current execution, not ask the repository what the latest code is.

Immutable artifacts are artifacts that are never modified in place after creation. For containers, an ECR image digest is stronger than a tag because the digest identifies the manifest content. Tags are useful labels but can be moved unless repository policy and process prevent it. For S3 bundles, use versioned buckets or commit-addressed keys such as releases/payment-api/7f3a91c1/appspec.yml. For Lambda, publish a version and move an alias rather than changing $LATEST directly in production.

Deployment strategies decide how much traffic sees the new artifact at one time. All-at-once replaces every target quickly; it is simple but has the largest blast radius. Rolling deployment replaces batches of instances or tasks; capacity planning matters because old and new versions coexist. Blue/green creates a separate replacement environment, then shifts traffic; rollback is fast if the old environment remains warm, but cost and database compatibility need planning. Canary exposes a small percentage first, waits for health evidence, then continues. Linear deployment moves traffic in equal steps over time. On AWS these patterns appear in CodeDeploy, ECS services, Lambda aliases, Elastic Beanstalk, CloudFormation change sets, and load balancer routing.

Configuration Anatomy

A delivery configuration has a few recurring parts. The source revision answers what change started the execution. The build specification answers how to test and package it. The artifact name or digest answers what exact thing will be promoted. The deployment group, ECS service, Lambda alias, or CloudFormation stack answers where it goes. Alarms and hooks answer when to stop. IAM roles answer who may do each action.

For CodeDeploy with ECS blue/green deployments, an AppSpec file maps the deployment to a task definition, container name, port, and optional lifecycle hooks. The service uses two target groups behind a load balancer. CodeDeploy registers the new task set, shifts listener traffic according to the selected schedule, monitors alarms and hook results, then either completes the shift or rolls traffic back to the previous task set.

Example 1: Build Once And Name The Artifact

The first example creates a container image and stores the deployment manifest under a commit-addressed S3 key. The expected behavior is that two different commits produce two different image tags and S3 keys. Re-running the same commit should produce the same names, which makes deployment logs traceable.

set -euo pipefail

APP_NAME=payment-api
COMMIT_SHA=$(git rev-parse --short=12 HEAD)
IMAGE_URI="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$APP_NAME:$COMMIT_SHA"
ARTIFACT_KEY="releases/$APP_NAME/$COMMIT_SHA/appspec.yml"

aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"
docker build --pull --label org.opencontainers.image.revision="$COMMIT_SHA" -t "$IMAGE_URI" .
docker push "$IMAGE_URI"
aws s3 cp appspec.yml "s3://$RELEASE_BUCKET/$ARTIFACT_KEY" --metadata commit="$COMMIT_SHA" --no-progress
printf 'IMAGE_URI=%s
ARTIFACT=s3://%s/%s
' "$IMAGE_URI" "$RELEASE_BUCKET" "$ARTIFACT_KEY"

The important detail is that the image name includes the Git revision and the image also carries an OCI revision label. In production, prefer deploying by ECR image digest after the push because a digest cannot silently move to different content. The S3 key contains the same revision so the deploy stage can prove which AppSpec belonged to which build.

Example 2: Describe The Replacement Service

The second example is an ECS AppSpec fragment for CodeDeploy. It is a fragment because the task definition ARN is normally rendered by the pipeline after registering a task definition that references the new image digest.

version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: <TASK_DEFINITION_ARN>
        LoadBalancerInfo:
          ContainerName: payment-api
          ContainerPort: 8080
Hooks:
  BeforeAllowTraffic: validate-schema
  AfterAllowTraffic: smoke-test

Before traffic is sent to the replacement task set, BeforeAllowTraffic can run a Lambda hook that checks migrations, configuration, or schema compatibility. After traffic starts, AfterAllowTraffic can run a smoke test against the production listener. If either hook fails, CodeDeploy marks the deployment failed and can roll traffic back when automatic rollback is configured.

Example 3: Verify A Deployment With Signals

The third example checks deployment state and a load balancer error signal. The deterministic output of the first command is three tab-separated fields: status, creator, and deployment type. Typical successful output is Succeeded user BLUE_GREEN. The metric command returns recent 5xx sums; an empty result means no datapoints matched the time window, while zero values mean datapoints existed with no counted 5xx responses.

set -euo pipefail

DEPLOYMENT_ID="$1"
aws deploy get-deployment --deployment-id "$DEPLOYMENT_ID" --query 'deploymentInfo.[status,creator,deploymentStyle.deploymentType]' --output text
aws cloudwatch get-metric-statistics   --namespace AWS/ApplicationELB   --metric-name HTTPCode_Target_5XX_Count   --statistics Sum   --period 60   --start-time "$(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"   --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)"   --dimensions Name=LoadBalancer,Value="$ALB_FULL_NAME"   --query 'Datapoints[].Sum' --output text

This example deliberately verifies both control plane and data plane evidence. A deployment can say Succeeded while users still see errors caused by a dependency, bad configuration, or a compatibility problem. Post-deployment checks should include application-level smoke tests and business-specific counters, not only service status.

Design Choices And Trade-Offs

Use all-at-once for low-risk internal workloads where fast rollback is enough and spare capacity is limited. Use rolling deployments when the service can tolerate mixed versions and when capacity can absorb taking some targets out of service. Use blue/green when you need fast rollback, a separate validation environment, or load balancer based traffic shifting. Use canary or linear shifting when unknown production behavior is the main risk and you have meaningful alarms that detect the issue quickly.

Immutable artifacts trade convenience for auditability. Mutable tags such as latest are easy during development, but they make incident reconstruction difficult because the tag may no longer identify what ran. Commit-addressed artifacts and image digests require more plumbing, but they let you answer: what code ran, who approved it, what tests passed, and which environments received it?

Database changes are the hard edge of deployment strategy. Traffic shifting protects compute changes, but it does not magically roll back destructive schema updates. Prefer expand-and-contract migrations: add backward-compatible columns or tables, deploy code that writes both old and new shapes where necessary, migrate data, then remove old structures in a later release. This lets old and new application versions coexist during rolling, canary, or rollback windows.

Failure Modes And Troubleshooting

Symptom: a rollback deploys the wrong container image. Cause: the deployment used a mutable tag such as prod or latest. Diagnose: compare the task definition image value, ECR digest, pipeline execution ID, and Git commit recorded in build metadata. Correct: deploy by digest or commit-specific tag, prevent tag overwrites for release repositories, and store the artifact identity in pipeline output.

Symptom: a blue/green deployment succeeds but users receive intermittent 500 responses. Cause: old and new tasks are both serving traffic while the new code assumes a schema or configuration that only exists after a later step. Diagnose: filter application logs by version label, compare error rates per target group, and check migration timing. Correct: make the change backward compatible, add a pre-traffic validation hook, and split schema removal into a later deployment.

Symptom: CodeBuild passes locally but fails in the pipeline with access denied. Cause: the developer identity has permissions that the CodeBuild service role lacks, or the artifact bucket and KMS key policies do not trust the role. Diagnose: inspect the failed action details, CloudTrail event, service role policy, bucket policy, and key policy. Correct: grant the specific S3, ECR, KMS, CloudWatch Logs, and deployment permissions needed by the stage.

Symptom: a canary never advances. Cause: an alarm is too sensitive, missing data is treated as breaching, or the smoke test is checking a dependency unrelated to the release. Diagnose: open the alarm history, inspect datapoint timestamps, and run the hook manually with the same input event. Correct: align periods and evaluation windows with expected traffic, choose missing-data behavior intentionally, and keep hooks focused on release safety.

Security, Reliability, And Performance Implications

Pipeline roles should be split by responsibility. The source stage does not need production deploy rights, and the deploy stage does not need permission to change repository settings. Protect artifact buckets with encryption, bucket policies, versioning, and limited write access. Store secrets in Secrets Manager or Systems Manager Parameter Store and inject them at runtime; do not bake environment secrets into images because the same image should move through environments unchanged.

Reliability depends on health checks that reflect real readiness. An ECS task can be running while the application cannot serve requests. Use load balancer health checks, container health checks, startup grace periods, and smoke tests together. Performance also affects deployment safety: if a new version uses more CPU per request, a rolling deployment may overload the reduced old capacity. Watch saturation as well as error counts during traffic shifting.

Hands-On Lab: Immutable ECS Release Skeleton

Prerequisites: AWS CLI configured for a sandbox account, Docker, Git, an existing ECR repository, a versioned S3 bucket for release artifacts, and IAM permissions for ECR push, S3 put, CodeDeploy read, and CloudWatch read. Use a non-production account because the commands create or publish release artifacts.

  1. Set AWS_ACCOUNT_ID, AWS_REGION, and RELEASE_BUCKET in your shell.
  2. Confirm the working tree commit with git rev-parse --short=12 HEAD.
  3. Run the build example from this lesson to push a commit-named image and upload the AppSpec bundle.
  4. Record the printed IMAGE_URI and S3 artifact URI in your release notes or pipeline execution metadata.
  5. Render the AppSpec fragment with the task definition ARN that references the new image digest.
  6. Start a sandbox CodeDeploy deployment or, if you do not have the ECS service prepared, perform a dry review by checking that every placeholder has a value and every referenced resource exists.
  7. Run the verification example with the deployment ID and inspect both deployment status and recent load balancer 5xx counts.

Verification: the ECR repository contains an image for the current commit, the S3 bucket contains a release object under the commit-specific key, the deployment reports an expected status such as Succeeded or InProgress, and CloudWatch shows no new elevated 5xx pattern during the shift. Cleanup: delete the sandbox deployment group if you created one, remove unused test images according to repository retention policy, and delete the S3 release object if your lab bucket is not governed by lifecycle rules. For rollback, redeploy the previous task definition or shift traffic back to the previous target group through CodeDeploy.

Assessment Exercises

  1. A team deploys ECS services with the tag latest and says CloudTrail is enough for auditing. Explain what evidence is still missing during an incident and redesign the artifact identity.
  2. You need to release code that reads from a new database column. Design a two-release sequence that remains safe during rolling deployment and rollback.
  3. A canary shifts 10 percent of traffic and then rolls back because latency increased. List the signals you would inspect to decide whether the artifact, capacity, dependency, or alarm configuration caused it.
  4. Choose all-at-once, rolling, blue/green, canary, or linear deployment for a low-traffic internal API and for a payment authorization service. Defend both choices with blast radius, cost, and rollback reasoning.
  5. Write the minimum IAM capabilities a build role needs to build a Docker image, push it to ECR, upload an AppSpec to S3, and write build logs.

Summary

CI/CD on AWS is a controlled chain of evidence: source revision, build environment, immutable artifact, deployment target, traffic shift, health signal, and rollback route. The strongest designs build once, promote the same artifact, deploy with a strategy matched to risk, and make rollback compatible with data and configuration changes. When those pieces are explicit, releases become inspectable operations instead of guesses about what changed.