AWS from Zero to Production — Phase 8, Part 3 of 5 · See the full series roadmap
“Everything fails, all the time.”
That’s Werner Vogels, Amazon’s CTO — and it’s the most useful sentence in cloud architecture. Disks fail. Instances fail. Availability Zones have bad days. Deployments go wrong. Someone deletes the wrong thing. Occasionally, a whole Region has problems.
Reliability engineering isn’t about preventing failure. It’s about deciding, in advance, how much failure you can tolerate, building systems that absorb it — and proving they do before a real incident tests you.
What you’ll walk away with
- ⏱️ RTO and RPO — the two numbers that drive every decision
- 🧯 Four disaster recovery strategies, from backup and restore to active-active
- 🏢 Multi-AZ and multi-Region patterns
- 💥 Fault injection with AWS Fault Injection Service
- 🧪 AWS Resilience Hub and DR testing
- 📖 Runbooks and incident management
Two numbers that drive everything
Recovery Time Objective (RTO)
How long can you be down? The maximum acceptable time between failure and restoration of service.
Recovery Point Objective (RPO)
How much data can you lose? The maximum acceptable gap between the last recoverable data and the failure.
last backup failure service restored
| | |
--------------+--------------------+-------------------------+------>
<------- RPO -------> <---------- RTO ---------->
(data lost) (downtime)
Where the numbers come from
Not from engineers — from the business. Ask:
- What does an hour of downtime cost — in revenue, penalties, and trust?
- What does losing an hour of orders cost?
- Which systems actually need fast recovery? (Usually fewer than people think.)
Lower RTO and RPO cost more. The goal is the right numbers for each system — not the smallest ones.
Disaster recovery strategies
AWS describes four strategies, from cheapest and slowest to most expensive and fastest.
1. Backup and restore
- Back up data, and copy it to another Region.
- After a disaster, rebuild infrastructure from code and restore data.
- RPO: hours. RTO: hours to a day.
- Cost: lowest — you pay for backups only.
Infrastructure as Code (Phase 6) is what makes this realistic. Without it, “rebuild everything” takes weeks.
2. Pilot light
- Data is live-replicated to the recovery Region — like an RDS cross-Region read replica or DynamoDB global table.
- Core infrastructure exists but is switched off or scaled to zero — no running app servers.
- After a disaster, turn on and scale up the app tier, then switch traffic.
- RPO: minutes. RTO: tens of minutes to hours.
3. Warm standby
- A scaled-down but fully working copy of production runs in the recovery Region, with live data replication.
- After a disaster, scale up and switch traffic.
- RPO: seconds to minutes. RTO: minutes.
4. Multi-site active-active
- Full production in two or more Regions, all serving traffic.
- After a disaster, the healthy Regions absorb the traffic.
- RPO: near zero. RTO: near zero.
- Cost and complexity: highest — especially for data consistency.
Strategy RPO RTO Cost
Backup & restore hours hours to a day $
Pilot light minutes tens of minutes $$
Warm standby seconds-mins minutes $$$
Active-active near zero near zero $$$$
Active-passive vs. active-active
- Active-passive: one site serves traffic; the other waits. Simpler. Failover is a decision and an action, so it takes time — and it must be tested, or it won’t work when you need it.
- Active-active: all sites serve traffic all the time. Failover is just less traffic going to one site. Every site is constantly proven to work — but your data layer must handle writes in multiple places.
Multi-AZ patterns
For most workloads, multi-AZ is the reliability baseline — and it’s relatively cheap.
- Load balancers across at least two AZs.
- Auto Scaling groups and ECS services spread across AZs.
- RDS Multi-AZ or Aurora with replicas in other AZs.
- NAT gateways in each AZ, with each private subnet routed to its own AZ’s NAT gateway.
- S3, DynamoDB, SQS, and Lambda are multi-AZ by default.
Static stability
A system is statically stable if it keeps working through an AZ failure without needing to make changes — like launching new instances — at the moment of failure.
Need 4 tasks to serve peak traffic, across 2 AZs:
Not statically stable: 2 + 2 -> lose an AZ, you have 2, and must scale during the incident
Statically stable: 4 + 4 -> lose an AZ, you still have 4
During a large-scale event, the control plane you’d use to recover may be busy. Pre-provisioned capacity doesn’t need it.
Multi-Region patterns
Go multi-Region when RTO/RPO requirements, regulations, or a global user base justify the cost and complexity.
Building blocks:
- Data: Aurora Global Database, DynamoDB global tables, S3 Cross-Region Replication, and cross-Region backup copies.
- Traffic: Route 53 failover and latency routing with health checks (Phase 3), or AWS Global Accelerator.
- Infrastructure: the same IaC deployed to each Region.
- Failover control: Amazon Application Recovery Controller for routing controls and readiness checks.
Watch out: Many multi-Region designs fail during real events because of something forgotten in the second Region: a secret, a quota, a DNS record, an ECR image, or a certificate. Test the whole failover, not just the database.
Fault injection
You don’t know if your system survives failure until you cause failure — safely, on purpose.
AWS Fault Injection Service (FIS) runs controlled experiments:
- Stop or terminate EC2 instances, or stop ECS tasks.
- Inject CPU stress, memory pressure, or network latency and packet loss.
- Fail over an RDS or Aurora database.
- Disrupt network connectivity for a subnet — simulating an AZ problem.
- Pause cross-Region replication or throttle API calls.
Every experiment has stop conditions — CloudWatch alarms that halt it immediately if things go too far.
Hypothesis: If one AZ's ECS tasks stop, error rate stays below 1% and
p99 latency below 800 ms.
Experiment: Stop all orders tasks in us-east-1a.
Stop if: 5xx alarm fires.
Result: ...
Start in non-production, with small experiments, and build up.
AWS Resilience Hub
AWS Resilience Hub helps you define, assess, and track resilience:
- Define the application — from CloudFormation stacks, Terraform state, resource groups, or tags.
- Set a resilience policy — target RTO and RPO for different disruption types: application, infrastructure, AZ, and Region.
- Assess — Resilience Hub estimates whether your architecture can meet the policy, and explains why not.
- Recommend — suggested improvements, alarms, Standard Operating Procedures, and FIS experiments.
- Track — reassess after changes, and see resilience drift over time.
Disaster recovery testing
A DR plan that’s never been tested is a guess.
- Tabletop exercises: walk through a scenario on paper. Cheap, and it finds missing steps and owners.
- Restore tests: restore backups regularly and measure the time (Phase 5, Part 2).
- Failover drills: actually fail over — a database, an AZ, or a Region — and fail back.
- Game days: planned events where the team practices responding to injected failures in a realistic environment.
Measure real RTO and RPO during tests. Then fix the gaps.
Operational runbooks
A runbook is a step-by-step procedure for a specific operational task or alert:
Runbook: Orders API 5xx alarm
1. Check the dashboard: is traffic normal? Which status codes?
2. Check recent deployments (GitHub Actions). If one finished in the last 30 minutes,
roll back: run the "rollback" workflow.
3. Check ECS service events and stopped task reasons.
4. Check RDS: CPU, connections, Database Insights.
5. Check the AWS Health Dashboard for us-east-1.
6. If unresolved after 15 minutes, escalate to the on-call lead.
Good runbooks are:
- Linked from the alarm itself, so responders find them instantly.
- Tested and kept up to date after every incident.
- Automated where possible — with Systems Manager Automation (Phase 6) — so the runbook becomes a button.
Incident management
When something goes wrong, structure beats heroics.
During the incident
- Declare it early, with a severity level. It’s fine to downgrade later.
- Assign roles: an incident commander coordinates; others investigate; someone owns communication.
- Mitigate first. Roll back, fail over, or scale up — restore service, then find the root cause.
- Keep a timeline as you go.
- Communicate status on a regular schedule, to stakeholders and customers.
After the incident
Run a blameless postmortem:
- What happened, and when — the timeline.
- What was the impact?
- What were the contributing factors — technical and process?
- What went well, and what didn’t?
- Action items, with owners and due dates, to prevent recurrence or reduce impact.
Blameless doesn’t mean accountability-free. It means assuming people acted reasonably with the information they had, and fixing the system that let the mistake happen.
Common mistakes
- Choosing RTO/RPO without the business.
- Single-AZ “temporary” setups that become permanent.
- Not statically stable — relying on scaling during an AZ failure.
- Multi-Region plans never tested end to end.
- Fault injection without stop conditions.
- Runbooks that exist only in someone’s head.
- Postmortems that blame people instead of fixing systems.
Key takeaways
- RTO is how long you can be down; RPO is how much data you can lose. The business sets them.
- Backup and restore, pilot light, warm standby, and active-active trade cost for recovery speed.
- Multi-AZ with static stability is the baseline for production.
- Multi-Region adds protection — and complexity that must be tested.
- Use FIS to prove resilience and Resilience Hub to track it.
- Runbooks and blameless postmortems turn incidents into improvements.
Quick self-check
Test yourself before moving on:
- Your RPO is 5 minutes and you back up nightly. Do you meet it?
- Which DR strategy keeps a scaled-down but working copy of production running?
- What makes a system statically stable?
- What’s the first priority during an incident?
Answers
- No. You could lose up to 24 hours of data.
- Warm standby.
- It survives a failure without needing to make changes — like scaling up — during the failure.
- Mitigate — restore service — then investigate the root cause.
What’s next
In Part 4, we zoom out to AWS’s own framework for good architecture: the six pillars of the Well-Architected Framework, the Well-Architected Tool, and how to run a workload review.
What’s the most useful thing a postmortem ever taught your team? Share it in the comments.