AWS Learning Journey

Everything Fails, All the Time: Reliability Engineering on AWS

RTO and RPO, backup and restore, pilot light, warm standby, active-active, multi-AZ and multi-Region patterns, fault injection, AWS Resilience Hub, DR testing, runbooks, and incident management.

9 min read · 1623 words
AWSSite Reliability EngineeringDisaster RecoveryCloud ArchitectureDevOps
Everything Fails, All the Time: Reliability Engineering on AWS cover image

AWS from Zero to Production — Phase 8, Part 3 of 5 · See the full series roadmap


“Everything fails, all the time.”

That’s Werner Vogels, Amazon’s CTO — and it’s the most useful sentence in cloud architecture. Disks fail. Instances fail. Availability Zones have bad days. Deployments go wrong. Someone deletes the wrong thing. Occasionally, a whole Region has problems.

Reliability engineering isn’t about preventing failure. It’s about deciding, in advance, how much failure you can tolerate, building systems that absorb it — and proving they do before a real incident tests you.

What you’ll walk away with


Two numbers that drive everything

Recovery Time Objective (RTO)

How long can you be down? The maximum acceptable time between failure and restoration of service.

Recovery Point Objective (RPO)

How much data can you lose? The maximum acceptable gap between the last recoverable data and the failure.

         last backup            failure               service restored
              |                    |                         |
--------------+--------------------+-------------------------+------>
              <------- RPO -------> <---------- RTO ---------->
              (data lost)           (downtime)

Where the numbers come from

Not from engineers — from the business. Ask:

Lower RTO and RPO cost more. The goal is the right numbers for each system — not the smallest ones.


Disaster recovery strategies

AWS describes four strategies, from cheapest and slowest to most expensive and fastest.

1. Backup and restore

Infrastructure as Code (Phase 6) is what makes this realistic. Without it, “rebuild everything” takes weeks.

2. Pilot light

3. Warm standby

4. Multi-site active-active

Strategy           RPO            RTO              Cost
Backup & restore   hours          hours to a day   $
Pilot light        minutes        tens of minutes  $$
Warm standby       seconds-mins   minutes          $$$
Active-active      near zero      near zero        $$$$

Active-passive vs. active-active


Multi-AZ patterns

For most workloads, multi-AZ is the reliability baseline — and it’s relatively cheap.

Static stability

A system is statically stable if it keeps working through an AZ failure without needing to make changes — like launching new instances — at the moment of failure.

Need 4 tasks to serve peak traffic, across 2 AZs:
  Not statically stable:  2 + 2  -> lose an AZ, you have 2, and must scale during the incident
  Statically stable:      4 + 4  -> lose an AZ, you still have 4

During a large-scale event, the control plane you’d use to recover may be busy. Pre-provisioned capacity doesn’t need it.


Multi-Region patterns

Go multi-Region when RTO/RPO requirements, regulations, or a global user base justify the cost and complexity.

Building blocks:

Watch out: Many multi-Region designs fail during real events because of something forgotten in the second Region: a secret, a quota, a DNS record, an ECR image, or a certificate. Test the whole failover, not just the database.


Fault injection

You don’t know if your system survives failure until you cause failure — safely, on purpose.

AWS Fault Injection Service (FIS) runs controlled experiments:

Every experiment has stop conditions — CloudWatch alarms that halt it immediately if things go too far.

Hypothesis:  If one AZ's ECS tasks stop, error rate stays below 1% and
             p99 latency below 800 ms.
Experiment:  Stop all orders tasks in us-east-1a.
Stop if:     5xx alarm fires.
Result:      ...

Start in non-production, with small experiments, and build up.


AWS Resilience Hub

AWS Resilience Hub helps you define, assess, and track resilience:

  1. Define the application — from CloudFormation stacks, Terraform state, resource groups, or tags.
  2. Set a resilience policy — target RTO and RPO for different disruption types: application, infrastructure, AZ, and Region.
  3. Assess — Resilience Hub estimates whether your architecture can meet the policy, and explains why not.
  4. Recommend — suggested improvements, alarms, Standard Operating Procedures, and FIS experiments.
  5. Track — reassess after changes, and see resilience drift over time.

Disaster recovery testing

A DR plan that’s never been tested is a guess.

Measure real RTO and RPO during tests. Then fix the gaps.


Operational runbooks

A runbook is a step-by-step procedure for a specific operational task or alert:

Runbook: Orders API 5xx alarm

1. Check the dashboard: is traffic normal? Which status codes?
2. Check recent deployments (GitHub Actions). If one finished in the last 30 minutes,
   roll back: run the "rollback" workflow.
3. Check ECS service events and stopped task reasons.
4. Check RDS: CPU, connections, Database Insights.
5. Check the AWS Health Dashboard for us-east-1.
6. If unresolved after 15 minutes, escalate to the on-call lead.

Good runbooks are:


Incident management

When something goes wrong, structure beats heroics.

During the incident

After the incident

Run a blameless postmortem:

Blameless doesn’t mean accountability-free. It means assuming people acted reasonably with the information they had, and fixing the system that let the mistake happen.


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. Your RPO is 5 minutes and you back up nightly. Do you meet it?
  2. Which DR strategy keeps a scaled-down but working copy of production running?
  3. What makes a system statically stable?
  4. What’s the first priority during an incident?

Answers

  1. No. You could lose up to 24 hours of data.
  2. Warm standby.
  3. It survives a failure without needing to make changes — like scaling up — during the failure.
  4. Mitigate — restore service — then investigate the root cause.

What’s next

In Part 4, we zoom out to AWS’s own framework for good architecture: the six pillars of the Well-Architected Framework, the Well-Architected Tool, and how to run a workload review.

What’s the most useful thing a postmortem ever taught your team? Share it in the comments.


Share this article

LinkedIn post draft