KMS Encryption, Secrets Manager, and Parameter Store

AWS systems usually need two related but different protections: encrypt data and deliver sensitive configuration without exposing it in code, images, logs, or deployment scripts. AWS Key Management Service, AWS Secrets Manager, and AWS Systems Manager Parameter Store solve different parts of that problem. KMS protects cryptographic keys and performs cryptographic operations. Secrets Manager stores, versions, retrieves, and rotates secrets such as database passwords and API tokens. Parameter Store stores application configuration values and can also store encrypted strings for lower-complexity secret use cases.

The outcome for this lesson is practical: choose the right service, explain what happens during encryption and retrieval, write least-privilege access, troubleshoot common failures, and build a small working pattern suitable for applications in the AWS Cloud Engineering course.

How The Services Fit Together

KMS is the root of the encryption design. A KMS key is a regional logical key with policy, grants, aliases, rotation settings, and backing key material controlled by AWS or imported by the customer. For most application work you do not download the key. You ask KMS to encrypt, decrypt, or generate data keys, and KMS enforces authorization before using the key material.

Envelope encryption is the central mechanism. Instead of sending large objects to KMS, an application asks KMS for a data key. KMS returns a plaintext data key and an encrypted copy of that data key. The application uses the plaintext data key locally to encrypt the payload, discards the plaintext key, and stores the encrypted payload with the encrypted data key. To decrypt later, the application sends the encrypted data key to KMS, receives the plaintext data key if authorized, decrypts the payload locally, then discards the key again. Many AWS services do this envelope encryption for you when you select a customer managed KMS key.

Secrets Manager stores a secret as a named resource with versions. Version staging labels such as AWSCURRENT, AWSPREVIOUS, and AWSPENDING make rotation safe because clients can keep asking for the current version while a rotation Lambda creates and tests a new credential. Secrets Manager encrypts secret values at rest with KMS, integrates with IAM and resource policies, and records API activity in CloudTrail. It is usually the right choice when the value is a credential, must be rotated, needs cross-account sharing controls, or requires audit-friendly secret lifecycle management.

Parameter Store stores named parameters in a hierarchy such as /prod/payments/db/host. Parameters can be plain String, StringList, or encrypted SecureString. SecureString values use KMS for encryption. Parameter Store is often right for feature flags, endpoint names, AMI identifiers, and environment configuration. It can hold secrets, but it does not provide the same rotation workflow as Secrets Manager.

API Anatomy

A KMS request normally names a key by key ID, key ARN, alias, or alias ARN. A key policy is mandatory and defines who can administer or use the key. IAM identity policies can grant KMS permissions only if the key policy allows that path. Common data-plane permissions are kms:Encrypt, kms:Decrypt, kms:GenerateDataKey, and kms:DescribeKey. Conditions such as kms:ViaService and encryption context keys narrow usage to a service or workload.

A secret has a name or ARN, a secret value, optional JSON structure, staging labels, tags, resource policy, and KMS key. Applications call GetSecretValue and normally cache the result for a short time to reduce latency and cost. A parameter has a name, type, value, version, tier, and optional policies. Applications call GetParameter or GetParametersByPath, with WithDecryption required for SecureString plaintext retrieval.

Example 1: Encrypting One Value With KMS

This example shows direct KMS encryption. It is useful for small values such as a bootstrap token, but not for large application files. The important behavior is that only principals allowed to use the selected key can decrypt the ciphertext.

set -euo pipefail

KEY_ALIAS="alias/course-kms-demo"
PLAINTEXT="database-bootstrap-token"

aws kms encrypt \
  --key-id "$KEY_ALIAS" \
  --plaintext "$PLAINTEXT" \
  --encryption-context app=course-agent,env=dev \
  --query CiphertextBlob \
  --output text

The output is a base64 ciphertext blob. It changes between encryptions because KMS encryption is not deterministic. The encryption context is not secret, but it is authenticated metadata. A later decrypt request must supply the same context or KMS refuses to decrypt. That lets you bind ciphertext to a workload, tenant, or environment.

Example 2: Storing Configuration In Parameter Store

Use Parameter Store when applications need structured configuration names and simple retrieval. In this example, the host is plain configuration while the password is encrypted as a SecureString. Keeping both under the same path lets an instance role retrieve only the application subtree.

set -euo pipefail

APP_PATH="/dev/course-agent/payments"
KEY_ALIAS="alias/course-kms-demo"

aws ssm put-parameter \
  --name "$APP_PATH/db_host" \
  --type String \
  --value "payments.dev.internal" \
  --overwrite

aws ssm put-parameter \
  --name "$APP_PATH/db_password" \
  --type SecureString \
  --key-id "$KEY_ALIAS" \
  --value "replace-me-in-real-use" \
  --overwrite

aws ssm get-parameters-by-path \
  --path "$APP_PATH" \
  --with-decryption \
  --query 'Parameters[].{Name:Name,Type:Type}' \
  --output table

The deterministic part of the output is that two rows are returned: one named db_host with type String and one named db_password with type SecureString. The command intentionally queries names and types, not values, because secret values should not be printed during verification.

Example 3: Storing A Rotatable Secret

Secrets Manager is better for a database credential because the value is a secret with lifecycle. Store related fields as JSON so clients fetch one versioned secret and parse the fields they need.

set -euo pipefail

SECRET_NAME="dev/course-agent/payments/db"
KEY_ALIAS="alias/course-kms-demo"

aws secretsmanager create-secret \
  --name "$SECRET_NAME" \
  --kms-key-id "$KEY_ALIAS" \
  --secret-string '{"username":"app_user","password":"replace-me-in-real-use","engine":"postgres"}' \
  --query '{Name:Name,VersionId:VersionId}' \
  --output json

aws secretsmanager get-secret-value \
  --secret-id "$SECRET_NAME" \
  --query '{Name:Name,Stages:VersionStages}' \
  --output json

The first command creates the secret and returns its name and version identifier. The second command should show a version stage containing AWSCURRENT. In a real rotation workflow, a rotation Lambda creates a pending version, tests it against the database, and moves AWSCURRENT only after the new credential works.

Least-Privilege Access Pattern

A workload that reads one secret does not need broad KMS access. It needs permission to call Secrets Manager for that secret and KMS decrypt permission constrained to calls made through Secrets Manager in the same region. This IAM policy fragment shows the shape of that control.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOneSecret",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:dev/course-agent/payments/db-*"
    },
    {
      "Sid": "DecryptOnlyThroughSecretsManager",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/12345678-1234-1234-1234-123456789012",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

The trade-off is precision versus administration effort. Narrow policies take more planning, but they limit damage when an application role is misused. A broad kms:Decrypt permission can accidentally allow the same role to decrypt unrelated ciphertext that uses the same key.

Design Choices And Trade-Offs

Choose a customer managed KMS key when you need explicit key policy control, audit separation, aliases, grants, or the ability to disable access independently from the service resource. AWS managed keys reduce setup but give less control over policies and sharing. Use one key per security boundary, not one key per every secret by default; too many keys create policy and quota overhead without meaningful isolation.

Choose Secrets Manager when the value is a credential, rotation matters, or consumers need staged versions. Choose Parameter Store when the value is general configuration, hierarchical lookup is useful, and rotation orchestration is not required. For high-read applications, cache secrets and parameters in memory with a bounded time-to-live. Fetching on every request adds latency, increases API cost, and can turn a control-plane throttling event into an application outage.

Prefer environment-specific names and aliases such as alias/course-prod-app over hard-coded key IDs in application configuration. Aliases make rotation and replacement easier, but authorization still applies to the underlying key. For multi-account systems, define whether the producing account, consuming account, or central security account owns the key and secret resource policy.

Failure Modes And Troubleshooting

Symptom: an application receives AccessDeniedException from Secrets Manager or Parameter Store. Cause: the role may lack the service read permission, the KMS key policy may not allow decrypt, or a condition such as kms:ViaService may name the wrong region. Diagnose: check the caller identity, inspect the identity policy, inspect the key policy, and find the denied API call in CloudTrail. Correct: grant the smallest missing permission and keep KMS conditions aligned with the service and region actually used.

Symptom: KMS decrypt fails with an invalid ciphertext or incorrect encryption context error. Cause: the application supplied different encryption context on decrypt than on encrypt, or it is trying to decrypt data encrypted under a different key. Diagnose: compare the stored metadata, key ARN, and encryption context used by the encrypt and decrypt paths. Correct: persist non-secret encryption context metadata with the ciphertext and make decrypt code supply the same values.

Symptom: a newly rotated database secret breaks clients. Cause: rotation moved AWSCURRENT before the credential worked everywhere, clients cached the old password too long, or the database rejected the new user state. Diagnose: review rotation Lambda logs, secret version staging labels, database authentication logs, and client cache time-to-live. Correct: fix the rotation step, restore AWSCURRENT to the last working version if needed, and test rotation in a non-production environment with the same client behavior.

Symptom: configuration reads become slow or throttled. Cause: the application retrieves parameters or secrets on every request. Diagnose: inspect application traces, CloudWatch metrics, and service API error counts. Correct: add client-side caching, reduce lookup frequency, batch Parameter Store reads by path where appropriate, and fail closed for missing secrets rather than silently using defaults.

Hands-On Lab

Prerequisites: an AWS account, AWS CLI configured for a sandbox account, permission to create a KMS key, SSM parameters, and Secrets Manager secrets, and a region stored in AWS_REGION. Use disposable names and never use real production passwords in the lab.

  1. Create or identify a customer managed KMS key and alias named alias/course-kms-demo.
  2. Run Example 1 and save the ciphertext only in a temporary shell variable or scratch file.
  3. Create the Parameter Store values from Example 2 and verify that the table shows parameter names and types without printing the password.
  4. Create the Secrets Manager secret from Example 3 and verify that the current version has the AWSCURRENT staging label.
  5. Attach a role policy shaped like the least-privilege example to a test role, then confirm that the role can read the intended secret but cannot read a neighboring secret.

Verification should avoid exposing values. Use metadata queries, CloudTrail lookup, and application health checks. For cleanup, delete the lab parameters, schedule deletion of the lab secret with an appropriate recovery window or force-delete only in a disposable account, and schedule KMS key deletion only if no other lab resource uses the key.

Assessment Exercises

  1. An application stores database host, port, username, and password. Which fields belong in Parameter Store, which belong in Secrets Manager, and why?
  2. Explain why envelope encryption stores an encrypted data key beside encrypted data instead of storing the plaintext data key.
  3. A role has secretsmanager:GetSecretValue but still cannot read a secret encrypted with a customer managed key. Name two places you would inspect and what evidence you expect to find.
  4. Design an encryption context for tenant-specific ciphertext. What metadata would you include, and what operational problem could appear if that metadata changes?
  5. How would you reduce latency for a service that reads the same secret thousands of times per minute without weakening secret rotation too much?

Summary

KMS supplies controlled cryptographic operations, Secrets Manager manages sensitive values with versioning and rotation, and Parameter Store manages named configuration with optional encryption. The strongest designs separate configuration from credentials, constrain KMS use with policies and conditions, cache reads carefully, verify without printing secrets, and rehearse recovery from permission, context, rotation, and throttling failures.