AWS Learning Journey

Designing Systems That Don't Fall Over: Distributed Patterns on AWS

Stateless apps, horizontal scaling, caching, async processing, circuit breakers, retries, bulkheads, throttling, rate limiting, eventual consistency, distributed transactions, the saga pattern, and observability.

9 min read · 1706 words
Distributed SystemsSoftware ArchitectureAWSMicroservicesSite Reliability Engineering
Designing Systems That Don't Fall Over: Distributed Patterns on AWS cover image

AWS from Zero to Production — Phase 10, Part 2 of 4 · See the full series roadmap


The recommendations service gets slow. Not down — just slow. Every request to it now takes eight seconds instead of eighty milliseconds.

The product page calls it. So product page threads start waiting. Then the thread pool fills up. Then the product page stops responding. The load balancer marks it unhealthy. Traffic shifts to the remaining instances, which fill up faster. Within ten minutes, the whole site is down — because a non-essential feature got slow.

That’s a cascading failure, and it’s how most large outages actually happen. Not one big thing breaking, but small things spreading.

Distributed system patterns exist to stop the spread. This article covers the ones every cloud engineer should know.

What you’ll walk away with


Stateless applications

A stateless service keeps no client-specific data in memory between requests. Any instance can handle any request.

Move state out:

Why it matters:

Everything you deployed in Phases 4 through 7 was stateless for exactly this reason.


Horizontal scaling

Horizontal scaling needs stateless services, a load balancer, and something to decide when to scale: Auto Scaling target tracking, ECS service auto scaling, or Lambda’s built-in concurrency (Phase 4).

The bottleneck moves. Scale the app tier from 2 to 50 instances, and the database becomes the bottleneck. Plan for connection pooling (RDS Proxy), read replicas, caching, and queues.


Caching

A cache stores the result of expensive work so you don’t repeat it. Layers, from closest to the user:

Caching rules:


Asynchronous processing

Not everything has to happen during the request:

Synchronous:   POST /orders -> charge card -> reserve stock -> send email -> generate PDF -> 201
               (slow, and any failure fails the whole request)

Asynchronous:  POST /orders -> save order -> publish OrderPlaced -> 202 Accepted
               workers: charge card, reserve stock, send email, generate PDF

Do only what’s needed to accept the request reliably. Everything else goes to a queue or event bus (Part 1).


Timeouts

Every network call needs a timeout. Without one, a slow dependency holds your threads, connections, and memory indefinitely — that’s how the opening story happened.


Retries

Retries recover from transient failures — if done well:


Circuit breakers

A circuit breaker stops calling a dependency that’s failing, giving it time to recover — and failing fast instead of waiting:

CLOSED     calls flow normally; failures counted
   | failure rate > 50% over 30s
   v
OPEN       calls fail immediately (or return a fallback) for 60s
   | after cooldown
   v
HALF-OPEN  a few trial calls; success -> CLOSED, failure -> OPEN

In the opening story, a circuit breaker on the recommendations call would have opened within seconds. The product page would have shown “popular items” as a fallback — and stayed up.

Implement circuit breakers with libraries in your language — like resilience4j for Java or pybreaker for Python — or with a service mesh.

Graceful degradation

Decide which features are essential. When non-essential dependencies fail, degrade: show cached recommendations, hide the reviews widget, queue the email for later. Users rarely notice a missing widget; they always notice a broken checkout.


Bulkheads

On a ship, bulkheads divide the hull into compartments so one leak doesn’t sink everything. In software:

Cell-based architecture

At large scale, AWS itself uses cells: independent, complete copies of a service, each serving a subset of customers. A bad deployment or poison request affects one cell, not everyone. Shuffle sharding assigns each customer to a random combination of resources, so one problem customer rarely shares all of its resources with another.


Throttling and rate limiting

Protect services from being overwhelmed — by bugs, abusive clients, or legitimate spikes:

Respond with 429 Too Many Requests and a Retry-After header, so well-behaved clients back off.


Eventual consistency

In distributed systems, data copies — replicas, caches, search indexes, downstream services — are updated asynchronously. For a moment, they disagree. Eventually, they converge.

You see it everywhere:

Design for it:


Distributed transactions

In a monolith with one database, a transaction guarantees all-or-nothing. Across services — each with its own database — there’s no single transaction.

Two-phase commit exists, but it’s slow, fragile, and couples services tightly. It’s rarely used in cloud architectures.

Instead, use the saga pattern.


The saga pattern

A saga breaks a business transaction into local transactions, one per service. If a step fails, previous steps are undone with compensating actions:

Step                       Compensation (if a later step fails)
1. Create order (PENDING)  -> cancel order
2. Charge payment          -> refund payment
3. Reserve inventory       -> release inventory
4. Create shipment         -> (last step; nothing after it)
5. Mark order CONFIRMED

If inventory reservation fails at step 3: refund the payment, cancel the order — in reverse order.

Orchestrated saga with Step Functions

Step Functions is a natural saga orchestrator:

ChargePayment --(fail)--> CancelOrder --> Fail
     |
ReserveInventory --(fail)--> RefundPayment --> CancelOrder --> Fail
     |
CreateShipment --(fail)--> ReleaseInventory --> RefundPayment --> CancelOrder --> Fail
     |
ConfirmOrder --> Succeed

Each Catch routes to the right compensations. Execution history shows exactly what happened.

Saga rules


Observability for distributed systems

When one request touches ten services, logs from one service aren’t enough.


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. A slow, non-essential dependency is taking down your site. Which two patterns help most?
  2. Why should timeouts get shorter further downstream?
  3. What replaces distributed transactions across microservices?
  4. Why must saga compensations be idempotent?

Answers

  1. Timeouts and a circuit breaker — with a fallback for graceful degradation.
  2. A downstream call that outlasts the caller’s timeout wastes resources — the caller has already given up.
  3. The saga pattern — local transactions with compensating actions.
  4. They may run more than once after retries or failures.

What’s next

In Part 3, we take reliability to the architecture level: high availability and disaster recovery in practice — from single-AZ risks to multi-Region failover with Route 53, replication, and tested runbooks.

What’s the worst cascading failure you’ve lived through? Tell the story in the comments.


Share this article

LinkedIn post draft