AWS from Zero to Production — Phase 10, Part 1 of 4 · See the full series roadmap
Here’s how a lot of systems grow.
The orders service, after saving an order, calls the payment service. Then the inventory service. Then the email service. Then analytics. Then the loyalty program someone added last quarter.
Now the orders service knows about five other systems. If the email service is slow, checkout is slow. If analytics is down, orders fail. And every new feature means changing — and redeploying — the orders service.
Event-driven architecture flips this around. The orders service announces one fact — “an order was placed” — and moves on. Every interested service listens and reacts on its own. Nobody waits. Nobody needs to know who’s listening.
What you’ll walk away with
- 📣 Events vs. messages, and why the difference matters
- 🚌 Amazon EventBridge: buses, rules, filtering, archive, and replay
- 📬 Amazon SNS and Amazon SQS, standard and FIFO
- ☠️ Dead-letter queues and redrive
- 🔁 Idempotent consumers, retries, and backoff
- 🔧 EventBridge Pipes
- 🎻 Choreography vs. orchestration
Events vs. messages
They look similar — JSON flowing between services — but they mean different things.
An event
- A fact about something that already happened:
OrderPlaced,PaymentFailed,UserSignedUp. - Named in the past tense.
- The producer doesn’t know or care who consumes it.
- Zero, one, or many consumers can react.
A message (or command)
- A request for a specific receiver to do something:
SendEmail,ChargeCard,GenerateInvoice. - The sender expects it to be handled — usually by one consumer.
{
"source": "acme.orders",
"detail-type": "OrderPlaced",
"detail": {
"orderId": "A17",
"customerId": "C42",
"total": 84.20,
"currency": "EUR",
"placedAt": "2026-10-31T09:14:02Z"
}
}
Design tip: Put enough data in the event that most consumers don’t need to call back to the producer — but not the whole database row. And never include secrets.
Why event-driven?
- Loose coupling: producers don’t depend on consumers being up — or even existing.
- Independent scaling: each consumer scales on its own.
- Resilience: a slow or failed consumer doesn’t break the producer.
- Extensibility: add a new consumer without touching the producer.
The costs
- Eventual consistency: consumers catch up “soon,” not instantly.
- Harder debugging: a request becomes a chain of asynchronous hops. You need tracing and correlation IDs.
- Duplicates and ordering: you must design for both.
Amazon EventBridge
EventBridge is a serverless event bus: producers put events on a bus; rules match events and route them to targets.
- Default bus: receives events from AWS services — an EC2 state change, an S3 object created, a CodePipeline failure.
- Custom buses: for your own application events.
- Partner buses: events from SaaS providers.
Rules and filtering
{
"source": ["acme.orders"],
"detail-type": ["OrderPlaced"],
"detail": {
"total": [{ "numeric": [">=", 500] }],
"currency": ["EUR", "GBP"]
}
}
This rule matches only large EUR and GBP orders — and sends them, say, to a fraud review Step Functions workflow. Patterns support prefixes, suffixes, numeric ranges, anything-but, and existence checks.
- Each rule can have multiple targets: Lambda, SQS, SNS, Step Functions, ECS tasks, API destinations (any HTTP endpoint), another event bus — even in another account.
- Input transformers reshape the event for each target.
Reliability features
- Retries with backoff, for up to 24 hours by default.
- A dead-letter queue per target for events that can’t be delivered.
- Archive and replay: keep events and replay them later — to recover a consumer after a bug, or to backfill a new consumer.
- Schema registry: discover event schemas and generate code bindings.
Amazon SNS: publish and subscribe
Amazon SNS pushes each message to every subscriber of a topic — Lambda, SQS, HTTP endpoints, email, SMS, and mobile push.
- Subscription filter policies let each subscriber receive only what it cares about — filtering on message attributes or the message body.
- FIFO topics preserve order and deduplicate, and deliver to SQS FIFO queues.
- Very high throughput, simple fan-out.
EventBridge or SNS?
- EventBridge: rich content filtering, many target types, AWS service events, SaaS events, archive and replay, schema registry. The default for application events.
- SNS: very high throughput, simple fan-out, push to mobile, SMS, and email, and FIFO ordering to SQS FIFO.
Many architectures use both.
Amazon SQS: queues
SQS (Phase 4) buffers messages until a consumer processes them. In event-driven systems, a queue in front of each consumer is a powerful pattern:
EventBridge rule ---> SQS queue ---> Lambda / ECS worker
(buffer, retry, DLQ)
The queue absorbs bursts, lets the consumer process at its own pace, and holds messages safely if the consumer is down.
Standard vs. FIFO
Standard queues:
- Nearly unlimited throughput
- At-least-once delivery — occasional duplicates
- Best-effort ordering
FIFO queues:
- Ordering within a message group — like all messages for
orderId=A17 - Deduplication within a five-minute window, by deduplication ID or content hash
- Lower throughput limits than standard — with a high-throughput mode for more
Rule of thumb: Use standard queues with idempotent consumers by default. Use FIFO when order truly matters per entity — like applying account balance updates in sequence.
Dead-letter queues
Some messages fail no matter how many times you retry: malformed data, a missing record, a bug.
A dead-letter queue (DLQ) catches them after maxReceiveCount attempts, so they don’t block the queue or retry forever:
- Alarm on DLQ depth (
ApproximateNumberOfMessagesVisible > 0). - Inspect the messages, fix the cause.
- Redrive them back to the source queue — SQS has built-in DLQ redrive.
EventBridge targets, SNS subscriptions, and asynchronous Lambda invocations all support DLQs or failure destinations too. Use them everywhere.
Event filtering
Filter as early as possible, so consumers don’t pay to receive and discard events:
- EventBridge rule patterns — at the bus.
- SNS subscription filter policies — per subscriber.
- Lambda event source mapping filters — for SQS, Kinesis, and DynamoDB Streams, filtered events never invoke your function.
Event replay
Things go wrong. A consumer had a bug for six hours and processed events incorrectly. With replay, you fix the bug and process those events again:
- EventBridge archives store events — all of them, or filtered — for a retention period you choose, and replay them to a bus.
- Kinesis and Kafka let consumers rewind to an earlier position (Phase 9).
- Events in the data lake — via Firehose — give you replay forever.
Replay only works safely if consumers are idempotent.
Idempotent consumers
You will receive duplicates: from at-least-once delivery, retries, replays, and redrives. An idempotent consumer produces the same outcome whether a message arrives once or five times.
Techniques:
- Natural idempotency: “set status to SHIPPED” is safe to repeat; “add 10 points” is not.
- Idempotency keys: record each processed event ID — for example, a DynamoDB conditional write — and skip ones you’ve seen.
- Upserts by business key instead of inserts.
import time
import boto3
from botocore.exceptions import ClientError
table = boto3.resource("dynamodb").Table("processed-events")
def handle(event):
try:
table.put_item(
Item={"eventId": event["id"], "ttl": int(time.time()) + 7 * 86400},
ConditionExpression="attribute_not_exists(eventId)",
)
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
return # already processed
raise
award_loyalty_points(event["detail"])
Watch out: This simple version records the event before doing the work, so if
award_loyalty_pointsfails, a retry will skip it. Production code should mark the key as in progress, then completed — and clear it on failure. Powertools for AWS Lambda has an idempotency utility that handles all of this for you.
Retry and backoff
When a call fails, retry — but carefully:
- Exponential backoff: wait 100 ms, 200 ms, 400 ms, 800 ms…
- Jitter: add randomness, so a thousand clients don’t retry at the same instant.
- Cap attempts and total time.
- Retry only retryable errors: timeouts, throttling, and 5xx — not validation errors.
AWS SDKs retry with backoff and jitter by default. Queues and EventBridge add retries at the infrastructure level. Be careful not to multiply them — SDK retries inside Lambda retries inside queue retries can turn one failure into hundreds of attempts.
EventBridge Pipes
EventBridge Pipes connect a source to a target with optional filtering and enrichment — no glue code:
Source Filter Enrich Target
DynamoDB -> only INSERTs -> Lambda / API / -> EventBridge bus,
Stream Step Functions SQS, Step Functions, ...
Sources include SQS, Kinesis, DynamoDB Streams, Amazon MQ, and Kafka. A classic use: turn DynamoDB changes into clean domain events on a bus — the transactional outbox pattern, without writing a poller.
Choreography vs. orchestration
Two ways to coordinate a multi-step business process — like order fulfillment.
Choreography
Each service reacts to events and emits its own. No central coordinator.
OrderPlaced -> Payment service -> PaymentSucceeded -> Inventory service
-> InventoryReserved -> Shipping service -> OrderShipped
- Pros: loose coupling, independent teams, easy to add reactions.
- Cons: the overall flow is implicit — spread across services — and harder to see, change, and debug.
Orchestration
A central workflow tells each service what to do, and tracks state.
Step Functions: charge payment -> reserve inventory -> create shipment
(on failure: refund payment, release inventory)
- Pros: the flow is explicit and visible, with built-in error handling and timeouts.
- Cons: the orchestrator is a central dependency, and services are more coupled to it.
In practice
Use choreography between domains — orders, billing, shipping — and orchestration within a domain for complex processes. Phase 10, Part 2 covers the saga pattern, which uses both.
Common mistakes
- Events as remote procedure calls — “event” names like
SendEmailNowthat expect one specific reaction. - Huge events with entire database rows — or secrets.
- Non-idempotent consumers.
- No DLQs or DLQ alarms.
- FIFO everywhere “just in case,” limiting throughput for no reason.
- Retry storms from stacked retries without jitter.
- No correlation IDs, making it impossible to trace one order through the system.
Key takeaways
- Events are facts about the past; messages are requests to act.
- EventBridge routes events with rich filtering, archive, and replay; SNS fans out; SQS buffers.
- Put a queue with a DLQ in front of consumers that need protection and retries.
- Expect duplicates — make every consumer idempotent.
- Retry with exponential backoff and jitter, without stacking retries.
- Choreograph between domains; orchestrate within them.
Quick self-check
Test yourself before moving on:
- Is
GenerateInvoicean event or a command? - A new team wants to react to every order. What changes in the orders service?
- When should you choose an SQS FIFO queue?
- A consumer had a bug for six hours. How can you reprocess those events?
Answers
- A command — it asks for something to be done.
- Nothing. They add a rule or subscription to the existing
OrderPlacedevents. - When ordering per entity matters, and you need deduplication.
- Replay them from an EventBridge archive — with idempotent consumers.
What’s next
In Part 2, we go deeper into distributed system design: statelessness, caching, circuit breakers, bulkheads, throttling, eventual consistency, distributed transactions, and the saga pattern.
Choreography or orchestration — which does your team prefer? Tell me in the comments.