cerf
Route 53 ARC multi-Region failover

Route 53 ARC multi-Region failover

Multi-Region resilience with Route 53 Application Recovery Controller

DNS is the switch that moves traffic between Regions; ARC is what lets you flip it when a Region is on fire.

The failover problem: DNS is the switch, but what flips it?

A two-Region disaster recovery design lives or dies on one question: what actually moves customer traffic from the primary Region to the standby Region? In an AWS active/standby or active/active architecture the answer is almost always DNS. Route 53 holds the authoritative records for your application hostname, and changing which endpoint those records resolve to is how you shift traffic between Regions. The topology diagram traces this: a client asks a recursive resolver, the resolver queries the Route 53 hosted zone, and Route 53 returns either the primary Region's load balancer or the standby Region's load balancer depending on record health.

Because the switch is DNS, the two variables that govern how fast a failover takes effect are the record TTL and how quickly the health signal changes. AWS recommends low TTLs (60 or 120 seconds are common) on records involved in failover so resolvers re-query soon after a change. Even with a low TTL, clients with open keep-alive connections keep hitting the old Region until they reconnect, which is why AWS also recommends lowering the ALB HTTP client keepalive duration (default 3600 seconds) toward your recovery time objective.

Route 53 decides record health with a fleet of health checkers distributed around the world. It considers an endpoint healthy when more than 18% of those checkers report it healthy, a deliberately low bar chosen so that a network partition isolating a few checker locations does not falsely mark a live endpoint down. Health checks run on a 30-second or 10-second interval with a configurable consecutive failure threshold. This is the machinery that plain DNS failover leans on, and understanding its limits is what motivates ARC.

Route 53 ARC multi-Region failoverActive/standby two-Region failover with ARC routing controls
Active/standby two-Region failover with ARC routing controlsGlobal DNSApplication Recovery Controllerus-east-1 (PRIMARY)us-west-2 (STANDBY)1. query app.example.com2. Route 53 authoritative answerprimary record health checksecondary record health checkreflects On/Offreflects On/Offhosts control panelgroupsgroupsUpdateRoutingControlState (5 endpoints)3. returns Primary ALBHTTPSQL writesafter failover: returns Standby ALBHTTPSQLAurora Global DB replicationClient /browserapp.example.comRecursive DNSresolvercaches per TTL (60s)Route 53 hostedzonefailover records + …Routing controlhealth checkreflects PRIMARY st…Routing controlhealth checkreflects STANDBY st…ARC routingcontrol clusterdata plane in 5 Reg…Control panel +safety rulesassertion: >=1 cont…Routingcontrol:…State: OnRoutingcontrol:…State: OffDR operator /automationflips the switchALB (Primary)us-east-1App tier (ASG)us-east-1Aurora GlobalDBwriter (us-east-1)ALB (Standby)us-west-2App tier (ASG)us-west-2 (pre-prov…Aurora GlobalDBreader / standby (u…
Trace
Route 53 failover records point at ARC routing control health checks; flipping a routing control moves traffic from the primary Region to the standby Region.

Route 53 DNS failover, and where plain health checks fall short

Route 53 gives you two failover shapes. Active-passive uses the failover routing policy: you create a primary and a secondary record with the same name and type, and Route 53 serves the primary while its health check passes, switching to the secondary only when all primary resources are unhealthy. Active-active uses any non-failover policy, such as weighted or latency, where every healthy record is a candidate and Route 53 simply stops returning a record when its health check fails.

For AWS endpoints you can create alias records for (ALB, NLB, CloudFront, S3 website endpoints), the recommended pattern is to set Evaluate Target Health to Yes rather than attaching an explicit health check, so Route 53 follows the alias target's own health. For non-alias targets, such as an EC2 web server or an on-premises host, you associate a Route 53 health check with each record. A calculated (parent) health check can combine up to 255 child checks when you need composite logic.

The weakness is subtle. Endpoint health checks answer 'can a TCP connection be opened and does the endpoint return 2xx/3xx,' not 'is this Region healthy enough to serve production.' During a partial or gray impairment, the load balancer can still pass its probe while dependencies behind it degrade, so automatic failover never triggers. Conversely, a transient blip can flap the check and fail you over when you did not want it. Plain health-check failover couples the decision to move traffic to whether an endpoint answers a probe, and breaking that coupling is exactly what a controlled recovery needs.

ARC routing controls: a deliberate switch decoupled from endpoint reachability

ARC routing controls replace 'is the endpoint answering?' with 'have we decided this Region should take traffic?'. A routing control is a simple On/Off switch. It is paired with a special kind of Route 53 health check, a routing control health check, whose status reflects the switch state rather than any probe of your endpoint. You point your failover or weighted records at those health checks, so flipping a routing control from On to Off makes Route 53 stop returning that Region's record, and flipping the other Region's control to On makes Route 53 start returning it.

The components stack up as a cluster, one or more control panels, the routing controls themselves, and the routing control health checks. In the topology diagram, PRIMARY is On and STANDBY is Off in steady state, so Route 53 returns the us-east-1 ALB. A DR operator or automation flips STANDBY to On and PRIMARY to Off, the two routing control health checks change status, and Route 53 begins returning the us-west-2 ALB. The application tiers and the Aurora global database were already provisioned and replicating in the standby Region; ARC only moves the traffic.

This decoupling is the point. The failover is now an explicit, auditable action you control, not a side effect of a probe. You can fail over during a gray failure that a health check would miss, and you will not flap on a transient blip. The tradeoff is that ARC does not decide for you when a Region is unhealthy; you, or your CloudWatch alarms and automation, must make that call and then execute the switch.

Route 53 ARC multi-Region failoverActive/standby two-Region failover with ARC routing controls
Active/standby two-Region failover with ARC routing controlsGlobal DNSApplication Recovery Controllerus-east-1 (PRIMARY)us-west-2 (STANDBY)1. query app.example.com2. Route 53 authoritative answerprimary record health checksecondary record health checkreflects On/Offreflects On/Offhosts control panelgroupsgroupsUpdateRoutingControlState (5 endpoints)3. returns Primary ALBHTTPSQL writesafter failover: returns Standby ALBHTTPSQLAurora Global DB replicationClient /browserapp.example.comRecursive DNSresolvercaches per TTL (60s)Route 53 hostedzonefailover records + …Routing controlhealth checkreflects PRIMARY st…Routing controlhealth checkreflects STANDBY st…ARC routingcontrol clusterdata plane in 5 Reg…Control panel +safety rulesassertion: >=1 cont…Routingcontrol:…State: OnRoutingcontrol:…State: OffDR operator /automationflips the switchALB (Primary)us-east-1App tier (ASG)us-east-1Aurora GlobalDBwriter (us-east-1)ALB (Standby)us-west-2App tier (ASG)us-west-2 (pre-prov…Aurora GlobalDBreader / standby (u…
Trace
Route 53 failover records point at ARC routing control health checks; flipping a routing control moves traffic from the primary Region to the standby Region.

The data plane is the whole point: flipping the switch when a Region is down

An ARC cluster is not a single endpoint. Each cluster is a data plane of endpoints in five AWS Regions, and this is the single most important design fact about the service. You change routing control states with the data plane API, UpdateRoutingControlState (and read them with ListRoutingControls), by connecting to any one of those five Regional endpoints. If one endpoint is unavailable, you retry against another. AWS's guidance is to pick an endpoint at random and have retry logic that can walk all five.

The reason this matters is the control plane / data plane split. The ARC control plane, meaning the console and the configuration APIs that create clusters, control panels, and routing controls, is not guaranteed to be available during a Regional impairment, and AWS explicitly says it should never be in your failover critical path. The extremely reliable data plane is engineered to stay available so you can still flip the switch when things are on fire. The reliability diagram shows the operator loading long-lived DR IAM credentials from a vault, calling a cluster endpoint, and that state change propagating through the control panel and routing controls to the Route 53 records.

This drives concrete operational practices. Bookmark or hard-code your five Regional cluster endpoints and routing control ARNs rather than relying on a DescribeCluster call (itself a control-plane operation) at failover time. Keep purpose-built long-lived IAM credentials in a secure vault so you are not dependent on a federated login flow that may itself be impaired, and drive failover from the API or CLI rather than the console. The always-on cluster carries an hourly cost, which is the price of having a switch that works during the exact event you built it for.

Route 53 ARC multi-Region failoverARC data plane vs control plane during a Regional impairment
ARC data plane vs control plane during a Regional impairmentARC data plane cluster (5 Regional endpoints) extremely reliable; use for failoverControl panel + safety rulesARC control plane (config, not failover path)Route 53 DNS failoverload DR credentialsUpdateRoutingControlStateretry if endpoint downretry if endpoint downavoid during eventmodels app (recovery group / cells)apply state changevalidated byvalidated byset Offset Onstate -> health checkstate -> health checkprimary record now unhealthysecondary record now healthyreturns Standby endpointconnect to Standby RegionDR operator /automationexecutes the runbookLong-lived IAMDR credentialskept in a vaultClusterendpointRegion 1ClusterendpointRegion 2ClusterendpointRegion 3ClusterendpointRegion 4ClusterendpointRegion 5ARC controlplane / consoleconfig APIsReadiness checkquotas, capacity, r…Control panelAssertion rule>=1 control On (no …Gating rulemaster on/off switchRouting controlPRIMARYOn -> OffRouting controlSTANDBYOff -> OnRoute 53failover…app.example.comRC health check(primary)RC health check(standby)Clientsnew lookups after T…Standby RegionALBus-west-2
Trace
Failover is driven through the five-Region data plane with long-lived credentials; safety rules validate the change before it reaches Route 53.

Guardrails and the rest of ARC: safety rules, readiness checks, zonal shift

Because routing controls are switches, the dangerous mistake is turning both Regions Off (a fail-open where traffic goes nowhere) or letting buggy automation reroute traffic at the wrong time. ARC safety rules guard against this. An assertion rule enforces that a condition holds whenever you change state, for example that at least one routing control in the set is On at all times, so you can never dark out the whole application. A gating rule creates a master switch: a gating routing control that must be On before any of its target routing controls can change, which lets you freeze automation during maintenance. If a safety rule blocks an update you genuinely need, you can explicitly override it.

Readiness check is the preparedness side of ARC. You model the application as a recovery group made of cells (typically one per Region), attach resource sets, and ARC continually audits that the standby matches production on quotas, capacity, and routing configuration. Critically, readiness check is not a health signal and must not be used as a failover trigger; it tells you whether the standby is ready to receive traffic, not whether the primary is broken. Note that readiness check is now closed to new customers, so new designs lean on Region switch, ARC's newer orchestration for automated multi-Region recovery across accounts.

Zonal shift solves a different scope. It temporarily moves traffic away from a single impaired Availability Zone to healthy AZs in the same Region for supported, opt-in resources such as load balancers, with an expiry from one minute up to three days (72 hours) that you can extend. Zonal autoshift lets AWS start that shift for you based on its internal telemetry. Zonal shift is about AZ-level recovery inside one Region and needs no routing-control cluster; multi-Region failover is the routing-control story. Knowing which tool matches which blast radius, AZ versus Region, is exactly what tasks 1.1 and 3.1 test.

Route 53 ARC multi-Region failoverARC data plane vs control plane during a Regional impairment
ARC data plane vs control plane during a Regional impairmentARC data plane cluster (5 Regional endpoints) extremely reliable; use for failoverControl panel + safety rulesARC control plane (config, not failover path)Route 53 DNS failoverload DR credentialsUpdateRoutingControlStateretry if endpoint downretry if endpoint downavoid during eventmodels app (recovery group / cells)apply state changevalidated byvalidated byset Offset Onstate -> health checkstate -> health checkprimary record now unhealthysecondary record now healthyreturns Standby endpointconnect to Standby RegionDR operator /automationexecutes the runbookLong-lived IAMDR credentialskept in a vaultClusterendpointRegion 1ClusterendpointRegion 2ClusterendpointRegion 3ClusterendpointRegion 4ClusterendpointRegion 5ARC controlplane / consoleconfig APIsReadiness checkquotas, capacity, r…Control panelAssertion rule>=1 control On (no …Gating rulemaster on/off switchRouting controlPRIMARYOn -> OffRouting controlSTANDBYOff -> OnRoute 53failover…app.example.comRC health check(primary)RC health check(standby)Clientsnew lookups after T…Standby RegionALBus-west-2
Trace
Failover is driven through the five-Region data plane with long-lived credentials; safety rules validate the change before it reaches Route 53.

Sources

Practice what you just read

6 questions from this architecture

Loading…