Slack
Slack's cellular architecture, or how to walk away from an availability zone
One flaky network link between AZs in us-east-1 convinced Slack that surviving zone failures means being able to drain a zone in minutes, not diagnose it in hours.
The gray failure that multi-AZ did not save them from
On June 30, 2021, a network link connecting availability zones in AWS us-east-1 started faulting intermittently. Slack was already deployed across multiple AZs, exactly the way the textbooks say, and users still saw slowness and degraded connections. The postmortem vocabulary matters here: this was a gray failure, meaning different components had different views of the availability of the system. Servers inside the impacted AZ saw their local backends as completely healthy while backends in other zones looked unavailable, and servers outside the zone saw the mirror image. When health checks disagree with each other, automated remediation has nothing solid to act on.
Slack's conclusion was to stop trying to automatically diagnose gray failures and instead build a blunt, reliable, human-operated lever: drain the whole zone. Their stated design goals were to remove as much traffic as possible from an AZ within 5 minutes, to do it without user-visible errors, to make drains and undrains incremental, and to guarantee the draining mechanism does not rely on resources in the AZ being drained. That last goal is the AWS static-stability principle in miniature: the recovery path must not depend on the thing that is failing.
The architecture that makes the lever safe to pull is siloing. Slack treats each service as N virtual services, one per AZ. A siloed service only receives traffic from within its AZ and only sends traffic upstream to servers in its AZ, so each zone behaves like a self-contained cell and a failure in one AZ is contained to that AZ. Once nothing meaningful crosses zone boundaries mid-request, removing a zone from service becomes a pure traffic-management decision at the front door.
The June 2021 gray failure: a regional call graph crossing an impaired AZ linkThe drain, step by step
Follow a normal request first. A Slack client lands on the edge, where Slack has migrated from HAProxy to the Envoy and xDS ecosystem, so every edge load balancer is an Envoy proxy. Envoy holds a weighted target cluster per availability zone and picks one; from that moment the request stays inside the chosen cell. The app frontend it hits calls only backends in the same AZ, those backends call their own in-AZ upstreams, and the response goes back out. Under healthy conditions the zones simply share load, each cell running the full service stack.
Now the drain. An operator decides AZ 3 looks wrong and triggers a drain. Rotor, Slack's in-house xDS control plane, pushes new weights to every edge Envoy through RTDS, the Envoy runtime discovery service, telling them to reweight their per-AZ target clusters. Propagation through the control plane is on the order of seconds, and weights support gradual drains with a granularity of 1 percent, so an operator can bleed a suspect zone slowly or zero it instantly. In-flight requests complete, new requests land only in healthy cells, and because the control signal originates at the edge, none of it depends on anything running inside the drained zone.
Why drain at the edge instead of teaching every internal service to shift traffic? Because Slack's services are written in Hack, Go, Java, and C++, and discover each other through the Envoy xDS API, the Consul API, and even DNS. A per-service drain would mean reimplementing the logic in every language and every discovery mechanism, and maintaining internal forks of open-source systems along the way. Concentrating the mechanism at the edge means one implementation covers everything siloed behind it. The documented exception is stateful storage: Vitess, Slack's main datastore, offers strongly consistent semantics, and each shard has a single primary that must accept all writes for that shard. If a frontend's local AZ does not hold the primary, the write crosses zones, and if a primary becomes unavailable, writes to that shard fail until it returns or a replica is promoted. Consistency, not laziness, is what keeps the data tier from being fully siloed.
Draining a zone at the edge: Envoy weights, Rotor/RTDS, and the Vitess exceptionWhat this teaches for the exam
For task 1.3, Slack's per-AZ weighted clusters are a self-built version of controls that ELB gives you off the shelf. Cross-zone load balancing is the exact knob: with it disabled, each load balancer node distributes traffic only to targets in its own AZ, which is Slack-style zonal affinity; with it enabled, every node sprays across all zones and you lose fault isolation in exchange for evenness. NLB disables cross-zone by default, ALB enables it by default and lets you turn it off per target group. The managed equivalent of Slack's drain lever is zonal shift in Amazon Application Recovery Controller: for supported resources such as ALBs and NLBs, you shift traffic away from an impaired AZ to the healthy zones in the same Region, with an expiry from one minute up to three days, and AWS tells you to prescale before shifting because the surviving zones absorb the displaced load. Slack's 5-minute goal and incremental weights are the same idea with more granularity.
For task 3.1, this study is really about intra-Region communication patterns. The Amazon Builders' Library makes the mathematical case Slack lived through: if regional services call other regional services, a request has multiple chances to touch the impaired zone, but zone-local call graphs mean a request that starts in a healthy AZ stays healthy end to end. That is why AWS keeps foundational data-plane components like the NAT gateway strictly zonal. Two testable habits fall out. First, know which AWS resources are zonal versus regional, because AZ-independent designs are built from zonal pieces: subnets, NAT gateways, and load balancer nodes per AZ. Second, apply the static-stability rule to any recovery design the exam offers you: a failover or drain mechanism that depends on control-plane actions inside the failing zone, or on capacity you have not already provisioned, is the wrong answer. Slack wrote that requirement down explicitly, and so does AWS.
The managed equivalents: cross-zone-off ELB, zonal NAT, and ARC zonal shiftMore diagrams
Sources
Practice what you just read