High Availability and Disaster Recovery: Redundancy, Failover, RPO, RTO and Multi-Region

Pixly's team proudly reports "we have backups". Then a storage misconfiguration corrupts the primary database on a Saturday night. The backups exist, in the same account and region. Restoring 2 TB from them takes nine hours, the restore script has never been run end to end, and nobody is sure which backup predates the corruption. The site is down for most of Sunday, and some data from the last six hours is gone for good.

High availability and disaster recovery are about making those numbers (how long down, how much lost) deliberate decisions made in advance, rather than discoveries made during the worst night of the year. This article covers how to design redundancy that actually works, how to choose a disaster-recovery strategy from your objectives, and the operational habits that make recovery real.

• The difference between high availability and disaster recovery.
• Failure domains, and why redundancy must cross them.
• Redundancy and failover patterns, and availability arithmetic with correlated failures.
• Error budgets: how much unavailability to plan for.
• RPO and RTO, and the four disaster-recovery strategies.
• Multi-region designs and how writes are handled.
• Backups that restore, and why replication is not a backup.
• Deployments and configuration changes as availability risks.

You need both. A multi-zone active system has excellent HA and can still lose data permanently to one bad DELETE statement, because the replicas copy it instantly.

A failure domain is a set of components that can fail together because they share something: power, network, hardware, software, configuration or people.

Redundancy protects only against failures in domains the copies do not share. Two servers in the same rack do not survive a rack failure; two regions running the same buggy release do not survive the bug. Good design asks, for each critical component: which domain failures does this redundancy cover, and which does it not?

Cloud providers expose availability zones, separate data centres in one region with independent power and networking, so that applications can spread across them. Amazon's S3 documentation, for example, describes zones as physically separated by a meaningful distance while within 100 km of each other, and S3 stores objects across at least three of them for most storage classes (Data protection in Amazon S3). Spreading across zones is the default way to achieve high availability within a region.

• N+1, N+2: run one or two more instances than the load requires, so losing some does not overload the rest (article 07's capacity with a zone lost).
• Active-active: all copies serve traffic. Failover is just "stop sending to the broken one". Requires the copies to share or synchronise state.
• Active-passive: one copy serves, another waits. Failover means promoting the passive copy (article 15). Simpler for stateful components; failover takes time and must be tested.
• Stateless tiers (app servers) are easy to make active-active across zones. Stateful tiers (databases) usually run a primary with a synchronous standby in another zone and automatic failover.

Static stability is worth designing for: the Amazon Builders' Library describes keeping enough pre-provisioned capacity in each zone that the loss of one zone requires no new resources to be launched, because the systems that launch resources may themselves be impaired during the event (Static stability using Availability Zones).

Article 02 showed that redundancy multiplies failure probabilities if failures are independent:

One zone at 99.9%: unavailable 0.1% of the timeThree zones, any one sufficient, independent: 0.001^3 = 1 in a billion unavailable

Real systems never achieve that, because of correlated failures: a shared dependency (one database primary, one DNS provider, one configuration service), a release deployed to all zones, an operator mistake. The real availability is dominated by these shared causes. When you estimate, include them:

Zone-redundant app tier:         ~ very high (independent zones)x one database primary (with standby, ~minute failover): 99.95% (assumption)x one global config service:     99.95% (assumption)x deployment-caused incidents:   99.9%  (assumption)= about 99.8%

The lesson: after zone redundancy, the shared components and change processes decide your availability. That is where to invest next.

Google's SRE book argues that 100% reliability is the wrong target for almost everything: users cannot tell the difference beyond a point (their own networks and devices fail more often), and each extra nine costs far more. It proposes setting an availability target and treating the remainder as an error budget that can be spent on releasing changes faster (Embracing Risk). If the target is 99.9% a month, about 43 minutes of failure is acceptable; when the budget is used up, the team slows releases and focuses on reliability.

For design, the point is: choose the target from the business need, then design to it. A payment API and an internal reporting tool deserve different targets and different costs.

Two numbers define what "recovered" means:

• Recovery Point Objective (RPO): the maximum acceptable data loss, measured in time. RPO of 5 minutes means losing at most the last 5 minutes of writes.
• Recovery Time Objective (RTO): the maximum acceptable downtime before service is restored.

        last good copy          disaster              service restored-----------|-----------------------X-------------------------|---------->           <------- RPO -------->   <---------- RTO ---------->           (data written here may be lost)   (users cannot use the system)

Both come from the business: what does an hour of downtime cost, and what does losing an hour of orders cost? Then they drive the technical strategy.

AWS's disaster-recovery whitepaper describes four broad strategies, from cheapest and slowest to most expensive and fastest (Disaster recovery options in the cloud). The ideas apply to any provider or your own data centres.

Important details from the whitepaper:

• Infrastructure as code is essential for every strategy: without it, rebuilding in another region is slow and error-prone.
• Pilot light versus warm standby: pilot light cannot serve requests until you switch on and scale compute; warm standby can serve immediately at reduced capacity.
• Prefer data-plane operations for failover: control planes (the APIs that create and configure resources) are typically less available than data planes, so a recovery plan that depends on launching lots of new resources during a regional event is fragile.
• Even active-active needs backups: for data corruption or deletion, the recovery point will be before the problem was discovered, and replication will have copied the damage to every region.
• Test regularly, or you do not know whether the strategy works.

Running in several regions improves latency for distant users and survives regional outages, but it forces you to decide how writes work. The AWS whitepaper names three approaches (Disaster recovery options in the cloud):

From article 19: write-local is AP behaviour with conflict resolution; write-global is CP for writes; write-partitioned avoids conflicts by giving every piece of data one owner. Data residency rules (some data must stay in a country) often push designs towards write-partitioned.

Traffic steering between regions uses DNS-based routing (with the TTL caveats from article 03), anycast or global load balancers. Decide in advance whether failover is automatic (fast, but a false alarm moves traffic unnecessarily and costs some data and availability) or manually triggered with automated steps ("push one button"). The AWS whitepaper notes that many organisations prefer manually initiated, automated failover for regional events for this reason.

Replication protects against losing machines. Backups protect against losing data: deletion, corruption, bugs, attackers. A sound backup strategy:

1. Point-in-time recovery for databases: base backups plus continuously archived change logs, so you can restore to any moment, such as one minute before the bad migration. PostgreSQL documents this as continuous archiving and point-in-time recovery (PostgreSQL: Continuous Archiving and PITR).
2. Isolation: keep copies in another region and another account or security boundary, so that compromised credentials or an account-wide mistake cannot delete both production and backups. The AWS whitepaper recommends cross-account backups for this reason.
3. Immutability: use write-once or object-lock features so backups cannot be altered or deleted for a period, which protects against ransomware.
4. Back up more than data: infrastructure definitions, configuration, secrets (in a secure way), and machine images.
5. Test restores regularly, time them, and compare the time with your RTO. A backup you have never restored is a hope.
6. Know your retention: how far back can you go? Corruption is sometimes discovered weeks later.

A large share of outages in practice follow a deployment or a configuration change. Treat change safety as part of availability:

• Progressive rollouts: canary to a small percentage, then expand, watching error rates and latency (article 08's weighted routing).
• Automatic rollback when health metrics degrade.
• Feature flags to turn features off without redeploying.
• Zone-by-zone and region-by-region deployment, so a bad release does not hit every failure domain at once.
• Configuration changes treated like code: reviewed, versioned, rolled out gradually.

Objectives (from the business): viewing 99.95% per month; uploads 99.9%. Regional DR: RPO 1 minute, RTO 30 minutes. Data corruption: restore to any point in the last 14 days.

Within the region (high availability):

Across regions (disaster recovery): warm standby.

• Asynchronous database replica in a second region (RPO about the replication lag, normally well under a minute).
• Object storage replicated to the second region.
• A small API fleet running there behind its own load balancer, receiving synthetic test traffic continuously, so the team knows it works.
• Infrastructure as code for scaling it to full size.
• Failover: an engineer triggers an automated runbook (promote the regional replica, scale the fleet, switch traffic), rehearsed quarterly. Target 30 minutes.

Why not active-active? It would cost roughly double, and require a multi-region write strategy for users, photos, likes and follows, with conflicts to handle. Pixly's objectives (30-minute RTO for a rare regional event) do not justify that today. The design records what would change the decision: a requirement for sub-minute regional RTO, or a large user base in a distant region needing low write latency.

Backups: daily base backups and continuous log archiving, copied to a separate backup account in another region with a 14-day immutability lock; monthly restore drills that measure restore time.

• Saying "multi-region" without stating RPO, RTO or a write strategy.
• Treating replicas as backups.
• Assuming independent failures and quoting impossible nines.
• Never rehearsing failover or restores.
• Depending on control-plane operations (creating resources) during a regional failover.
• Deploying the same change everywhere at once.

• State the availability target and the RPO and RTO before choosing a strategy.
• Default to multi-zone within a region for HA; justify multi-region with requirements.
• Name the DR strategy (backup and restore, pilot light, warm standby, active-active) and why.
• If multi-region, state the write strategy and how conflicts are avoided or resolved.
• Mention tested, isolated backups with point-in-time recovery.

0.9995 (database) x 0.999 (auth) = about 0.9985, or 99.85%

• Next: Observability: Logs, Metrics, Traces, SLOs and Alerts, so you can see failures, measure availability and know when to fail over.
• Reference: the AWS Well-Architected reliability pillar and Google Cloud's architecture framework for reliability cover these practices in depth: Reliability pillar, Google Cloud reliability.