AWS from Zero to Production — Phase 10, Part 2 of 4 · See the full series roadmap
The recommendations service gets slow. Not down — just slow. Every request to it now takes eight seconds instead of eighty milliseconds.
The product page calls it. So product page threads start waiting. Then the thread pool fills up. Then the product page stops responding. The load balancer marks it unhealthy. Traffic shifts to the remaining instances, which fill up faster. Within ten minutes, the whole site is down — because a non-essential feature got slow.
That’s a cascading failure, and it’s how most large outages actually happen. Not one big thing breaking, but small things spreading.
Distributed system patterns exist to stop the spread. This article covers the ones every cloud engineer should know.
What you’ll walk away with
- 🧊 Stateless applications and horizontal scaling
- ⚡ Caching layers
- 📨 Asynchronous processing
- 🔌 Circuit breakers, retries, and timeouts
- 🚢 Bulkheads
- 🚦 Throttling and rate limiting
- ⏳ Eventual consistency
- 🔄 Distributed transactions and the saga pattern
- 🔭 Observability for distributed systems
Stateless applications
A stateless service keeps no client-specific data in memory between requests. Any instance can handle any request.
Move state out:
- Sessions → ElastiCache or DynamoDB, or signed tokens like JWTs.
- Uploaded files → S3, not local disk.
- Job progress → a database or Step Functions.
Why it matters:
- Scale out by adding instances — no sticky sessions or data migration.
- Replace failed instances freely — nothing is lost.
- Deploy with rolling or blue-green strategies safely.
Everything you deployed in Phases 4 through 7 was stateless for exactly this reason.
Horizontal scaling
- Vertical scaling: a bigger machine. Simple, but limited, and usually requires a restart.
- Horizontal scaling: more machines. Practically unlimited, and resilient — losing one of twenty instances barely matters.
Horizontal scaling needs stateless services, a load balancer, and something to decide when to scale: Auto Scaling target tracking, ECS service auto scaling, or Lambda’s built-in concurrency (Phase 4).
The bottleneck moves. Scale the app tier from 2 to 50 instances, and the database becomes the bottleneck. Plan for connection pooling (RDS Proxy), read replicas, caching, and queues.
Caching
A cache stores the result of expensive work so you don’t repeat it. Layers, from closest to the user:
- Browser and CDN — CloudFront caches static assets and cacheable API responses at the edge (Phase 3).
- API caching — API Gateway REST APIs can cache responses.
- Application cache — ElastiCache for Valkey or Redis (Phase 5).
- Database caches — DynamoDB Accelerator (DAX) for DynamoDB.
Caching rules:
- Cache what’s read often and changes rarely.
- Always set TTLs.
- Plan invalidation — and accept some staleness.
- Protect against stampedes: when a popular key expires, thousands of requests can hit the database at once. Use request coalescing, or refresh hot keys before they expire.
Asynchronous processing
Not everything has to happen during the request:
Synchronous: POST /orders -> charge card -> reserve stock -> send email -> generate PDF -> 201
(slow, and any failure fails the whole request)
Asynchronous: POST /orders -> save order -> publish OrderPlaced -> 202 Accepted
workers: charge card, reserve stock, send email, generate PDF
Do only what’s needed to accept the request reliably. Everything else goes to a queue or event bus (Part 1).
- Faster responses
- Absorbs bursts — queues buffer load spikes
- Isolates failures — a slow PDF service doesn’t slow checkout
Timeouts
Every network call needs a timeout. Without one, a slow dependency holds your threads, connections, and memory indefinitely — that’s how the opening story happened.
- Set timeouts based on the dependency’s normal latency — for example, a bit above its p99.
- Make them shorter the further upstream you are. If the user-facing request times out at 3 seconds, a downstream call can’t take 10.
- Remember all the layers: SDK clients, HTTP libraries, database drivers, load balancers, and API Gateway.
Retries
Retries recover from transient failures — if done well:
- Exponential backoff with jitter (Part 1).
- Limit attempts.
- Only retry idempotent operations — or use idempotency keys.
- Retry at one layer. Retries at every layer multiply: 3 retries × 3 layers = 27 attempts per user request.
- Use a retry budget — for example, retries may add at most 10% extra load. When a dependency is truly down, retrying everything just makes it worse.
Circuit breakers
A circuit breaker stops calling a dependency that’s failing, giving it time to recover — and failing fast instead of waiting:
CLOSED calls flow normally; failures counted
| failure rate > 50% over 30s
v
OPEN calls fail immediately (or return a fallback) for 60s
| after cooldown
v
HALF-OPEN a few trial calls; success -> CLOSED, failure -> OPEN
In the opening story, a circuit breaker on the recommendations call would have opened within seconds. The product page would have shown “popular items” as a fallback — and stayed up.
Implement circuit breakers with libraries in your language — like resilience4j for Java or pybreaker for Python — or with a service mesh.
Graceful degradation
Decide which features are essential. When non-essential dependencies fail, degrade: show cached recommendations, hide the reviews widget, queue the email for later. Users rarely notice a missing widget; they always notice a broken checkout.
Bulkheads
On a ship, bulkheads divide the hull into compartments so one leak doesn’t sink everything. In software:
- Separate connection pools or thread pools per dependency — a slow dependency can only exhaust its own pool.
- Separate services or queues for critical and non-critical work.
- Reserved Lambda concurrency per function, so one runaway function can’t starve the rest (Phase 4).
- Separate accounts per workload (Phase 1) — the ultimate bulkhead for quotas and blast radius.
Cell-based architecture
At large scale, AWS itself uses cells: independent, complete copies of a service, each serving a subset of customers. A bad deployment or poison request affects one cell, not everyone. Shuffle sharding assigns each customer to a random combination of resources, so one problem customer rarely shares all of its resources with another.
Throttling and rate limiting
Protect services from being overwhelmed — by bugs, abusive clients, or legitimate spikes:
- API Gateway throttling per stage, method, and usage plan — per-customer limits with API keys.
- AWS WAF rate-based rules — block IPs that send too many requests (Phase 3).
- Application-level limits with a token bucket — often backed by ElastiCache for shared counters.
- Load shedding: when overloaded, reject some requests quickly with
429or503rather than slowing down for everyone.
Respond with 429 Too Many Requests and a Retry-After header, so well-behaved clients back off.
Eventual consistency
In distributed systems, data copies — replicas, caches, search indexes, downstream services — are updated asynchronously. For a moment, they disagree. Eventually, they converge.
You see it everywhere:
- DynamoDB eventually consistent reads and GSIs
- RDS read replicas
- Caches and CDNs
- Every event-driven consumer (Part 1)
Design for it:
- Read your own writes from the primary when a user needs to see their change immediately.
- Show status honestly: “Your order is being processed.”
- Make operations commutative and idempotent where possible.
- Reconcile: periodic jobs that detect and fix drift between systems.
Distributed transactions
In a monolith with one database, a transaction guarantees all-or-nothing. Across services — each with its own database — there’s no single transaction.
Two-phase commit exists, but it’s slow, fragile, and couples services tightly. It’s rarely used in cloud architectures.
Instead, use the saga pattern.
The saga pattern
A saga breaks a business transaction into local transactions, one per service. If a step fails, previous steps are undone with compensating actions:
Step Compensation (if a later step fails)
1. Create order (PENDING) -> cancel order
2. Charge payment -> refund payment
3. Reserve inventory -> release inventory
4. Create shipment -> (last step; nothing after it)
5. Mark order CONFIRMED
If inventory reservation fails at step 3: refund the payment, cancel the order — in reverse order.
Orchestrated saga with Step Functions
Step Functions is a natural saga orchestrator:
ChargePayment --(fail)--> CancelOrder --> Fail
|
ReserveInventory --(fail)--> RefundPayment --> CancelOrder --> Fail
|
CreateShipment --(fail)--> ReleaseInventory --> RefundPayment --> CancelOrder --> Fail
|
ConfirmOrder --> Succeed
Each Catch routes to the right compensations. Execution history shows exactly what happened.
Saga rules
- Compensations must be idempotent — they may run more than once.
- Compensations aren’t rollbacks — a refund is a new transaction, and the customer may see both the charge and the refund.
- Order steps wisely — put the steps most likely to fail, or hardest to compensate, first.
Observability for distributed systems
When one request touches ten services, logs from one service aren’t enough.
- Correlation IDs: generate an ID at the edge, pass it through every call and event, and include it in every log line.
- Distributed tracing: AWS X-Ray or OpenTelemetry (with the AWS Distro for OpenTelemetry) show the path and timing of each request across services — so you can see which hop is slow.
- Service maps: CloudWatch Application Signals and X-Ray build maps of dependencies with latency and error rates.
- Structured logs with consistent fields across services (Phase 8).
- Per-dependency metrics: latency, errors, timeouts, and circuit breaker state for each downstream call.
Common mistakes
- No timeouts, or timeouts longer than the caller’s.
- Retries at every layer, with no jitter or budget.
- Treating every dependency as essential, so any failure becomes an outage.
- One shared connection pool for every dependency.
- Two-phase commit across microservices.
- Sagas without idempotent compensations.
- No correlation IDs or tracing.
Key takeaways
- Stateless services scale horizontally and recover easily.
- Cache in layers, with TTLs and stampede protection.
- Move non-essential work to asynchronous processing.
- Timeouts, careful retries, and circuit breakers stop cascading failures.
- Bulkheads and cells limit blast radius; throttling and load shedding protect capacity.
- Embrace eventual consistency, and use sagas with compensations instead of distributed transactions.
- Trace requests end to end with correlation IDs and distributed tracing.
Quick self-check
Test yourself before moving on:
- A slow, non-essential dependency is taking down your site. Which two patterns help most?
- Why should timeouts get shorter further downstream?
- What replaces distributed transactions across microservices?
- Why must saga compensations be idempotent?
Answers
- Timeouts and a circuit breaker — with a fallback for graceful degradation.
- A downstream call that outlasts the caller’s timeout wastes resources — the caller has already given up.
- The saga pattern — local transactions with compensating actions.
- They may run more than once after retries or failures.
What’s next
In Part 3, we take reliability to the architecture level: high availability and disaster recovery in practice — from single-AZ risks to multi-Region failover with Route 53, replication, and tested runbooks.
What’s the worst cascading failure you’ve lived through? Tell the story in the comments.