Direct Connect + VPN resiliency
Resilient hybrid connectivity: Direct Connect with VPN backup
One dedicated fiber is a single point of failure; BGP quietly parks a VPN behind it and swaps in when the fiber dies.
Direct Connect primary, VPN backup: the pattern
AWS Direct Connect (DX) gives you a dedicated, private link with consistent bandwidth and lower, more predictable latency than the internet. The catch is that a single DX connection is a single point of failure: a cut fiber, a failed router port, or an outage at the Direct Connect location takes it down, and AWS publishes no availability SLA for a lone connection. A Site-to-Site VPN over the public internet is the cheapest, fastest-to-provision way to add a second path: two IPsec tunnels terminating on a virtual private gateway (VGW) or a transit gateway (TGW). The design goal is that on-premises to VPC traffic rides DX in steady state and shifts to the VPN only when DX fails, then returns automatically once DX recovers.
The mechanism is BGP, not a health-check script. Both paths advertise the same on-premises prefixes into AWS and both learn the same VPC and Region prefixes back. For an identical destination prefix, AWS route selection prefers the route learned over Direct Connect, so the VPN sits idle as a hot standby carrying no data. When the DX BGP session drops, AWS withdraws its DX-learned routes, the VPN route becomes best, and traffic cuts over. When DX comes back and re-advertises, its route is preferred again and traffic returns. The diagram traces both the steady-state DX path and the failover VPN path to the same workload.
Do not confuse this with the separate "AWS Direct Connect + AWS Site-to-Site VPN" pattern in the VPC connectivity-options whitepaper, where an IPsec tunnel runs over a DX public VIF to encrypt the traffic that flows on Direct Connect itself. That pattern is about confidentiality on the DX path; here the VPN is a distinct, internet-based backup path for resiliency.
How BGP fails traffic over to the VPN, and back
The reason DX wins automatically is baked into AWS path selection. On a virtual private gateway, when prefixes are identical AWS prioritizes routes in this order, most to least preferred: BGP-propagated routes from a Direct Connect connection, then manually added static routes for the VPN, then BGP-propagated routes from the VPN. On a transit gateway the rule is longest-prefix-match first, and for equal prefixes Direct Connect is preferred over VPN. Critically, the health of a VPN tunnel endpoint takes precedence over other routing attributes, and this precedence applies on both VGW and TGW. Because DX outranks VPN for the same prefix, you get standby behavior for free as long as both paths advertise matching prefixes.
Failure-detection speed is what determines the outage length, and that is a design choice. Without Bidirectional Forwarding Detection (BFD), a dead DX link is only noticed when the BGP hold timer expires (default about 90 seconds), so traffic can black-hole for a minute and a half. Asynchronous BFD is automatically enabled on every Direct Connect virtual interface but does nothing until you also configure it on your customer router. AWS sets the BFD liveness interval to 300 ms with a multiplier of 3, so once configured, failure is detected in under a second and BGP tears the session down and withdraws routes, letting the VPN take over quickly.
Failback is automatic too. When the DX BGP session re-establishes, its routes are reinserted and, being higher priority, are immediately preferred again, so traffic returns to DX with no manual route-table or on-premises change. The main operational risk is a flapping DX link causing repeated cutovers; mitigate with BGP dampening or hold-down on the customer side rather than by disabling the backup.
Influencing path selection: local preference, more-specific routes, and the AS_PATH trap
A frequent exam trap: you cannot use AS_PATH prepending, local preference, or MED to make the VPN preferred over Direct Connect (or vice versa). Those attributes only break ties among attachments of the same type, DX versus DX or tunnel versus tunnel. Between a DX route and a VPN route for the same prefix, DX always wins regardless of how many times you prepend your ASN. If you truly need the VPN to carry a prefix while DX is up, you advertise a more specific prefix over the VPN so that longest-prefix-match, which is evaluated before any attribute, selects it.
For return traffic from AWS to on-premises across two Direct Connect connections, AWS evaluates in this order: longest prefix match, then local preference, then AS_PATH length, then MED. Local preference is expressed with the community tags 7224:7100 (low), 7224:7200 (medium), and 7224:7300 (high), and these are evaluated before AS_PATH. For an active/passive DX pair with equal prefix lengths, tag the primary VIF's prefixes 7224:7300 and the backup's 7224:7100; AWS then returns traffic over the high-preference location and fails to the other on withdrawal. For active/active load sharing, tag both with the same value (for example 7224:7200) and AWS uses ECMP.
Because longest-prefix-match overrides local preference, a more specific route silently defeats your community tags. This bites hybrid designs with a VPN on a transit gateway: the TGW can propagate more-specific VPC routes (individual /24s) out over the VPN to on-premises, and the on-premises router then prefers those specifics over a summarized /16 learned via DX, pulling traffic onto the backup unexpectedly. Control propagation and prefix advertisement deliberately, as the diagram highlights.
Higher-resiliency options: two locations, and DX + DX
The Direct Connect Resiliency Toolkit defines the higher tiers. Maximum Resiliency uses separate connections terminating on separate devices in more than one Direct Connect location, which protects against device, connectivity, and full-location failure and is the configuration eligible for the 99.99% SLA. High Resiliency uses one connection at each of two locations and targets 99.9%. A single-site redundant model (at least two connections on separate devices in one location) protects only against device failure, not a location outage, and is positioned for development and test. A Direct Connect gateway lets any of these reach VPCs across multiple Regions.
Where does VPN backup fit against DX + DX? A VPN is the cheapest way to add resiliency to a single DX: it adds a path that survives a DX-location or DX-device failure, but it rides the public internet with variable latency and loss, and it caps at 1.25 Gbps per IPsec tunnel. On a virtual private gateway only one tunnel is active at a time and ECMP is not supported; on a transit gateway you can ECMP across tunnels for up to 50 Gbps aggregate. For bandwidth-heavy or latency-sensitive workloads, a second Direct Connect (DX + DX at two locations) is the stronger choice, and it is the only way to reach the published DX SLAs.
The two are not mutually exclusive. A common resilient design is DX + DX at two locations for the SLA, plus a Site-to-Site VPN as a last-resort backup. Because the transit gateway automatically prefers DX-learned routes over the VPN, the VPN carries production traffic only in the rare event that both Direct Connect paths are down at once, as the third flow in the diagram shows.
Tradeoffs, cost, and testing the failover
Cost drives the DX-versus-VPN-backup decision. A Site-to-Site VPN is inexpensive: an hourly connection charge plus standard data-transfer, and it provisions in minutes. A second Direct Connect roughly doubles the fixed cost, port hours plus a cross-connect at a second colocation, but delivers dedicated bandwidth and the 99.99%/99.9% SLAs that a VPN cannot. VPN backup therefore buys meaningful resiliency at a fraction of a second DX, accepting a performance penalty only during the (hopefully rare) failover window.
Do not assume failover works because BGP is configured. Use the Resiliency Toolkit Failover Test, which disables the BGP session on a virtual interface to simulate a real failure, and confirm that the VPN takes over and that applications tolerate the higher-latency internet path. Instrument BGP session state and VPN tunnel state in CloudWatch so an operator is alerted the moment the standby path goes live, and re-run the test after any router or prefix change.
The recurring failures in production are predictable: prefixes that do not match between DX and VPN, so DX preference never engages; BFD left unconfigured on the router, so a fiber cut costs a 90-second outage; unintended more-specific route propagation over a TGW-attached VPN; and customer gateways that do not handle the asymmetric routing that failover can introduce. Design for all four and the pattern gives you sub-second, hands-off failover and automatic failback.
Sources
- Direct Connect routing policies and BGP communities - AWS Direct Connect User Guide
- Route tables and AWS Site-to-Site VPN route priority
- AWS Direct Connect Resiliency Recommendations
- AWS Direct Connect Resiliency Toolkit
- AWS Direct Connect + AWS Site-to-Site VPN - VPC Connectivity Options whitepaper
- Influencing traffic over hybrid networks using longest prefix match
Practice what you just read