API Gateway, EventBridge, SQS, SNS, and Step Functions

API Gateway, EventBridge, SQS, SNS, and Step Functions are the main AWS building blocks for serverless integration. In this lesson, the outcome is practical: design a request-to-workflow path where an HTTP request enters through API Gateway, becomes an event, is routed by EventBridge, buffered by SQS when needed, broadcast through SNS when fan-out is needed, and coordinated by Step Functions when the process has multiple durable steps.

These services matter in the Containers and Serverless section because serverless applications rarely consist of one function. Most useful systems need ingress, routing, buffering, fan-out, retries, state, and compensation. The design skill is knowing which managed service should own each responsibility instead of forcing one Lambda function or container worker to do everything.

Purpose and Outcome

API Gateway exposes HTTP APIs and REST APIs to clients. It accepts requests, applies authorization and throttling, validates or transforms payloads, and invokes an integration such as Lambda, an AWS service, or a private backend. EventBridge moves events between producers and consumers through event buses and rules. SQS stores messages in a queue until consumers are ready. SNS publishes one message to many subscribers. Step Functions runs a state machine, records execution history, and coordinates task, choice, wait, retry, catch, map, and parallel states.

A common pattern is an order intake flow. API Gateway receives POST /orders. The integration sends an OrderSubmitted event to EventBridge. EventBridge routes that event to a Step Functions state machine for payment and fulfillment, to an SQS queue for asynchronous warehouse work, and to an SNS topic for notifications. Each service has a narrow job: API Gateway handles the synchronous edge, EventBridge decides who should hear about an event, SQS absorbs back pressure, SNS fans out messages, and Step Functions records the long-running decision path.

How the Services Work Internally

API Gateway is the front door. A route such as POST /orders maps to an integration. With a Lambda proxy integration, API Gateway packages the method, path, headers, query parameters, and body into an event document. With an AWS service integration, API Gateway can call another AWS API directly, such as PutEvents on EventBridge. That removes Lambda from simple ingestion paths, but it also means request mapping templates, IAM roles, and response mappings become part of the API behavior.

EventBridge stores events on an event bus and evaluates rules against event envelopes. The important fields are source, detail-type, detail, account, region, time, and resources. Rules use event patterns to match values. When a rule matches, EventBridge invokes one or more targets. Targets can include Lambda, Step Functions, SQS, SNS, API destinations, and other buses. EventBridge is best for semantic routing: something happened, and independently owned consumers may react.

SQS is a pull-based buffer. Producers send messages to a queue. Consumers call ReceiveMessage, process messages, and then call DeleteMessage. When a message is received, it becomes invisible for the visibility timeout. If the consumer does not delete it before the timeout expires, the message becomes visible again. Standard queues maximize throughput and deliver at least once, which means duplicates are possible. FIFO queues preserve order within a message group and support deduplication, but with different throughput and design constraints.

SNS is push-based fan-out. A publisher sends one message to a topic, and SNS delivers it to subscriptions such as SQS queues, Lambda functions, HTTPS endpoints, email, SMS, or mobile push. SNS topic filtering can avoid delivering irrelevant messages to subscribers. SNS is a good fit when one event must be distributed to several independently managed subscribers and those subscribers do not need a shared queue.

Step Functions is a workflow engine. A state machine is written in Amazon States Language, a JSON-based definition. Each execution has input, state transitions, retries, catches, and output. Standard workflows are durable and keep detailed execution history for long-running business processes. Express workflows are optimized for high-volume, short-duration processing. Step Functions is strongest when you need visible orchestration: branch on payment result, retry a downstream task, wait for a callback, or compensate after a partial failure.

Configuration Anatomy

An API Gateway design starts with routes, integrations, authorizers, stages, throttling, and access logs. The integration role must be allowed to call the target AWS service, and the target resource must accept the request. For service integrations to EventBridge, the request body is transformed into entries for PutEvents.

An EventBridge design starts with the bus, event schema, rules, targets, retry policy, and dead-letter queue. A rule pattern should match stable business fields such as source and detail-type, not incidental text inside a payload. An SQS design starts with queue type, retention period, visibility timeout, redrive policy, encryption, and consumer concurrency. An SNS design starts with topic policy, subscriptions, filter policies, delivery status, and dead-letter handling. A Step Functions design starts with workflow type, input shape, states, retry and catch clauses, task permissions, logging, and execution naming.

Example 1: Match an EventBridge Event

The first example shows the smallest useful routing decision. The event pattern matches only order events from the checkout service whose status is SUBMITTED. The deterministic result of the final command is true, because the sample event satisfies every field in the pattern.

set -euo pipefail

cat > /tmp/order-pattern.json <<'JSON'
{
  "source": ["course.checkout"],
  "detail-type": ["OrderSubmitted"],
  "detail": {
    "status": ["SUBMITTED"]
  }
}
JSON

cat > /tmp/order-event.json <<'JSON'
{
  "source": "course.checkout",
  "detail-type": "OrderSubmitted",
  "detail": {
    "orderId": "ord-1001",
    "status": "SUBMITTED",
    "total": 42.50
  }
}
JSON

aws events test-event-pattern \
  --event-pattern file:///tmp/order-pattern.json \
  --event file:///tmp/order-event.json

This teaches the most important EventBridge habit: route on event metadata and stable business attributes. If the order status were DRAFT, the output would be false, and no target attached to that rule would run.

Example 2: Buffer Work with SQS

The second example sends a warehouse message to SQS and receives it with long polling. Long polling reduces empty receives by waiting briefly for a message to arrive. The receipt handle returned by ReceiveMessage is not the message ID; it is the token required to delete that specific delivery attempt.

set -euo pipefail

QUEUE_URL="https://sqs.us-east-1.amazonaws.com/111122223333/course-warehouse"

aws sqs send-message \
  --queue-url "$QUEUE_URL" \
  --message-body '{"orderId":"ord-1001","sku":"book-7","quantity":1}' \
  --output json

aws sqs receive-message \
  --queue-url "$QUEUE_URL" \
  --wait-time-seconds 10 \
  --max-number-of-messages 1 \
  --output json

The send response includes a MessageId and checksums. The receive response includes the body and a ReceiptHandle. A worker should process the message idempotently and call DeleteMessage only after durable success. If the worker crashes before deletion, SQS can deliver the same message again after the visibility timeout.

Example 3: Fan Out with SNS to SQS

The third example uses SNS for fan-out and SQS for reliable subscriber delivery. One notification is published to a topic, and subscribed queues receive their own copies. This is different from several workers reading one SQS queue: with SNS fan-out, each subscriber gets an independent delivery path.

set -euo pipefail

TOPIC_ARN="arn:aws:sns:us-east-1:111122223333:course-order-events"

aws sns publish \
  --topic-arn "$TOPIC_ARN" \
  --subject "Order submitted" \
  --message '{"orderId":"ord-1001","event":"OrderSubmitted"}' \
  --message-attributes '{"eventType":{"DataType":"String","StringValue":"OrderSubmitted"}}' \
  --output json

The publish response contains a MessageId. If a subscription has a filter policy for eventType = OrderSubmitted, it receives the message. A subscription filtering for PaymentFailed does not. Use SNS when the publisher should not know how many subscribers exist or whether one subscriber is a queue, a Lambda function, or an HTTPS endpoint.

Example 4: Coordinate Steps with Step Functions

The fourth example is a complete Step Functions definition. It checks an order total, approves small orders automatically, and routes larger orders to manual review. The expected behavior is deterministic: input with {"total":42.5} ends in AutoApproved; input with {"total":2500} ends in ManualReviewRequired.

{
  "Comment": "Course order approval workflow",
  "StartAt": "CheckTotal",
  "States": {
    "CheckTotal": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.total",
          "NumericLessThanEquals": 1000,
          "Next": "AutoApproved"
        }
      ],
      "Default": "ManualReviewRequired"
    },
    "AutoApproved": {
      "Type": "Pass",
      "Result": {
        "status": "APPROVED"
      },
      "End": true
    },
    "ManualReviewRequired": {
      "Type": "Pass",
      "Result": {
        "status": "REVIEW_REQUIRED"
      },
      "End": true
    }
  }
}

This example shows why Step Functions is not just a trigger mechanism. The state machine owns the process state and records the path taken. A Lambda function could implement the same if statement, but Step Functions makes the workflow visible, retryable, and auditable as the process grows.

Design Choices and Trade-Offs

Choose API Gateway when clients need an HTTP contract. Choose EventBridge when producers and consumers should be decoupled by event meaning. Choose SQS when work must wait until consumers have capacity. Choose SNS when a publisher must push one message to many subscribers. Choose Step Functions when the business process has multiple steps, branches, waits, retries, or compensating actions.

Do not use EventBridge as a queue. It routes events but consumers do not pull from a backlog the way they do with SQS. Do not use SNS when subscribers need to coordinate consumption of the same work item; an SQS queue with competing consumers is usually better. Do not hide a complex workflow inside one Lambda function simply because it starts small. Once operators need to answer which step failed, which retry is pending, or which compensation ran, Step Functions provides a better operational model.

The main trade-offs are latency, durability model, ordering, cost shape, and operational visibility. Synchronous API Gateway calls are easier for clients but tie response time to downstream behavior. Asynchronous events improve resilience but require idempotency and a way for clients to learn final status. FIFO queues help with ordering but require careful message group design. Step Functions improves workflow clarity but adds state transition cost and a definition that must be versioned with application logic.

Failure Modes and Troubleshooting

Symptom: API clients receive 403 or 500 when submitting an order. Cause: the API Gateway integration role cannot call events:PutEvents, or the mapping template creates an invalid request. Diagnose: check API Gateway execution logs, access logs, and the integration response body. Confirm the IAM role policy and test the transformed request. Correct: grant the integration role only the needed bus permission and fix the request mapping so each entry has Source, DetailType, Detail, and EventBusName.

Symptom: an EventBridge rule never invokes its target. Cause: the event pattern does not match the actual event, often because detail-type or nested detail fields differ. Diagnose: capture a real event, run aws events test-event-pattern, inspect rule metrics, and confirm the target permission. Correct: adjust the pattern to stable fields and add a dead-letter queue for target delivery failures.

Symptom: SQS messages are processed repeatedly. Cause: the worker exceeds the visibility timeout, crashes before deletion, or does not treat duplicate deliveries as normal. Diagnose: compare processing duration with visibility timeout, inspect approximate receive count, and review dead-letter queue redrive settings. Correct: make processing idempotent, increase or extend visibility timeout, delete only after success, and configure a redrive policy.

Symptom: a Step Functions execution fails after a downstream timeout. Cause: the task state lacks a retry or catch path for a known transient failure. Diagnose: inspect execution history and identify the failing state, error name, and input. Correct: add targeted Retry rules for transient errors and Catch transitions for business failures that require compensation or review.

Security, Performance, and Reliability

Use IAM roles per integration or worker instead of broad shared credentials. API Gateway should use an authorizer or IAM authentication when the route is not public, and stages should have throttling to protect downstream services. EventBridge bus policies should control which accounts or services may publish events. SQS and SNS resource policies should restrict publishers and subscribers. Encrypt queues and topics when payload sensitivity requires it, and avoid putting secrets in event bodies because events are often copied to logs, queues, and multiple consumers.

Reliability depends on idempotency. EventBridge, SQS, and SNS all require consumers to tolerate retries or duplicate messages. Include a stable business identifier such as orderId and store processing results so duplicate deliveries do not repeat irreversible work. For performance, buffer spiky workloads with SQS, tune consumer concurrency, and use batch receives where appropriate. For workflows, keep Step Functions state input small and pass references to large objects stored in S3 or DynamoDB.

Hands-On Lab

Prerequisites: an AWS account or sandbox, AWS CLI configured for a non-production account, permission to create API Gateway, EventBridge, SQS, SNS, IAM, Lambda, and Step Functions resources, and a unique name prefix such as course-events-yourinitials.

  1. Create an SQS queue named with your prefix and configure a dead-letter queue with a small maximum receive count for testing.
  2. Create an SNS topic named with your prefix. Subscribe the SQS queue to the topic and update the queue policy so the topic can send messages to it.
  3. Create an EventBridge rule that matches source = course.checkout and detail-type = OrderSubmitted. Add the SNS topic as a target.
  4. Create a simple Step Functions state machine using the approval definition from this lesson. Start one execution with {"total":42.5} and another with {"total":2500}.
  5. Optional edge path: create an API Gateway route that calls EventBridge PutEvents through an AWS service integration. Send a test request and confirm an event reaches the rule.

Verification: use aws events test-event-pattern before creating the rule, publish a matching event, then receive a message from the SQS queue. Confirm that Step Functions execution history shows the expected terminal state for each input. Also publish a non-matching event and verify that no new queue message appears.

Cleanup: delete the API route if created, the EventBridge rule and targets, the Step Functions state machine, the SNS subscription and topic, the SQS queue and dead-letter queue, and any IAM roles created for the lab. Cleanup matters because queues, logs, and executions can retain test payloads.

Assessment Exercises

  1. An order API must respond in under 300 milliseconds, but warehouse processing can take minutes. Which service should separate the client response from warehouse work, and what idempotency key would you include?
  2. A payment event must notify billing, analytics, and customer email systems without the payment service knowing those subscribers. Would you use SQS alone, SNS, or EventBridge? Explain the routing model.
  3. A worker sometimes takes longer than expected and the same SQS message is processed twice. What metrics and fields would you inspect, and what two corrections would you make?
  4. A workflow has three steps, and the second step may fail after the first step charged a customer. Why is Step Functions a better fit than a single hidden function, and where would you put retry versus compensation?
  5. An EventBridge rule pattern includes too many nested payload details and stops matching after a harmless schema change. Redesign the event envelope fields used for routing.

Summary

API Gateway, EventBridge, SQS, SNS, and Step Functions solve different integration problems. API Gateway accepts HTTP traffic, EventBridge routes events by meaning, SQS buffers pull-based work, SNS pushes fan-out notifications, and Step Functions coordinates durable workflows. Strong serverless designs use these services together with clear event schemas, narrow permissions, idempotent consumers, dead-letter handling, and verification that covers both matching and failure paths.