AWS Learning Journey

When a Region Goes Dark: High Availability and Disaster Recovery in Practice

Single-AZ risks, multi-AZ deployment, cross-Region replication, database failover, Route 53 failover, backup strategies, recovery testing, choosing RTO and RPO, DR runbooks, and the real trade-off between cost and availability.

10 min read · 1839 words
AWSDisaster RecoveryHigh AvailabilityCloud ArchitectureSite Reliability Engineering
When a Region Goes Dark: High Availability and Disaster Recovery in Practice cover image

AWS from Zero to Production — Phase 10, Part 3 of 4 · See the full series roadmap


In Phase 8, you learned the theory: RTO, RPO, and four disaster recovery strategies.

This article is the practice. We’ll take the orders platform you’ve built across this series and ask, layer by layer: what happens if this fails? An instance. An Availability Zone. An entire Region. A bad deployment. Someone deleting the database.

Then we’ll decide — with numbers — how much protection is worth paying for.

What you’ll walk away with


Availability, in numbers

Availability is usually expressed in nines:

Availability   Downtime per year   Downtime per month
99%            ~3.65 days          ~7.3 hours
99.9%          ~8.8 hours          ~43.8 minutes
99.95%         ~4.4 hours          ~21.9 minutes
99.99%         ~52.6 minutes       ~4.4 minutes
99.999%        ~5.3 minutes        ~26 seconds

Dependencies multiply

If a request depends on components in series, their availabilities multiply:

Load balancer 99.99% × app 99.9% × database 99.95% ≈ 99.84%

The system is less available than its weakest part. Redundancy — components in parallel — is how you go up:

Two independent copies at 99.9% each:  1 − (0.001 × 0.001) = 99.9999%

That’s the whole idea behind multi-AZ: independent copies, so one failing doesn’t take the service down. (Real copies aren’t fully independent, so the real number is lower — but the principle holds.)


Single-AZ risks hiding in plain sight

Many architectures look multi-AZ on a diagram, but have a single-AZ dependency somewhere:

Exercise: For each component, ask: “If this AZ disappeared right now, what would break?” Write down every answer.


Multi-AZ, layer by layer

For the orders platform:

Layer             Multi-AZ setup
DNS               Route 53 (global, highly available by design)
Load balancer     ALB in subnets across 2-3 AZs
Compute           ECS tasks spread across AZs; enough capacity to lose one AZ
Outbound access   One NAT gateway per AZ, each private subnet routed to its own
Database          RDS Multi-AZ (or Aurora with a replica in another AZ)
Cache             ElastiCache with replicas and automatic failover across AZs
Queues / events   SQS, SNS, EventBridge (multi-AZ by default)
Object storage    S3 Standard (multi-AZ by default)
Secrets / config  Secrets Manager, Parameter Store (Regional services)

Most of this is configuration, not new architecture. Multi-AZ is the best value in availability: big protection for modest cost.


Cross-Region replication

For Regional disasters — or regulatory requirements — data must exist in a second Region. Everything the application needs to run must be there too.

Data

Everything else people forget

Watch out: Replication also replicates mistakes. A bad DELETE or a corrupting bug reaches the second Region in seconds. Cross-Region replication protects against Regional failure; backups — with point-in-time recovery and immutable vaults — protect against data corruption and deletion. You need both.


Database failover

What actually happens — and how long it takes:

Make applications failover-friendly


Route 53 failover

DNS is how users find the healthy Region (Phase 3):

api.example.com   failover PRIMARY    -> ALB us-east-1   health check: /health
api.example.com   failover SECONDARY  -> ALB us-west-2   health check: /health

Amazon Application Recovery Controller (ARC)

Automatic DNS failover can flap — failing over and back on noisy health checks — or fail over when it shouldn’t. ARC adds control:


Backup strategies

Backups protect against what replication can’t: deletion, corruption, ransomware, and bad deployments.


Recovery testing

Untested recovery is a guess (Phase 8). Build a testing calendar:

Monthly    Restore a database backup; run validation queries; record time
Quarterly  Force an RDS failover; zonal shift an ALB; measure impact
Twice/yr   Full Regional failover game day in staging; fail back
Ongoing    AWS Backup restore testing; ARC readiness checks; FIS experiments

Each test produces two numbers — actual RTO and actual RPO — and a list of things that went wrong. That list is the most valuable output.


Choosing RTO and RPO

Not every workload deserves the same protection. Tier them:

Tier   Example                     RTO          RPO          Strategy
1      Checkout, payments API      < 15 min     < 1 min      Warm standby / active-active
2      Orders admin, internal API  < 4 hours    < 15 min     Pilot light
3      Analytics, reporting        < 24 hours   < 24 hours   Backup and restore
4      Dev and test                days         days         Rebuild from IaC

Questions to set the tier:

Then check the cost of the strategy against the impact it prevents.


DR runbooks

A disaster recovery runbook is a tested, step-by-step plan:

Runbook: Regional failover — orders platform (Tier 1)

Trigger:  us-east-1 impaired > 15 min (Health Dashboard + alarms), approved by incident commander
Owner:    on-call platform engineer

1. Confirm the recovery Region is ready (ARC readiness checks green)
2. Fail over Aurora Global Database to us-west-2
3. Scale ECS service in us-west-2 to production capacity
4. Flip ARC routing control: us-east-1 OFF, us-west-2 ON
5. Verify: smoke tests, dashboards, error rates
6. Communicate: status page, stakeholders
7. Monitor; plan failback once us-east-1 recovers (reverse the steps)

Last tested: 2026-09-14 (staging) — actual RTO 22 min

Store runbooks where responders can find them during an outage — not only in a system hosted in the failed Region. Automate steps with Systems Manager Automation or Step Functions where you can.


Cost vs. availability

Every extra nine costs more — usually much more:

Worked example (illustrative):
  Downtime cost:            $20,000 per hour
  Expected Regional outage: ~4 hours every 2 years without DR  -> ~$40,000/year risk
  Warm standby cost:        ~$3,000/month                      -> $36,000/year

In this example, warm standby roughly breaks even on direct costs — before counting reputation and SLA penalties. For a Tier 3 workload with $500/hour impact, the same spend would make no sense.

Rule: Let the business impact decide the strategy — not the engineering team’s enthusiasm, and not fear.


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. Three components in series are each 99.9% available. Roughly what’s the system availability?
  2. Why isn’t cross-Region replication a substitute for backups?
  3. Name three things besides data that must exist in the recovery Region.
  4. What does an ARC routing control give you that automatic DNS failover doesn’t?

Answers

  1. About 99.7% (0.999³).
  2. Replication copies deletions and corruption too; backups let you go back in time.
  3. Any three of: container images, secrets, KMS keys, AMIs, certificates, configuration, IaC, and quotas.
  4. A deliberate, reliable switch — failover when a human decides, without flapping.

What’s next

In Part 4, we finish the architecture phase with money: cost and performance optimization — rightsizing, Savings Plans, Spot, storage tiers, NAT and data transfer costs, serverless and database tuning, Compute Optimizer, anomaly detection, and FinOps.

Has your team ever done a real Regional failover? How did it go? Tell me in the comments.


Share this article

LinkedIn post draft