cerf
Zoom

Zoom

Zoom: scaling real-time video during the 2020 surge

Thirty times the daily participants in four months, and the fix was regional entry points plus an EC2 fleet growing by thousands of instances a day.

Ten million to three hundred million

In December 2019 Zoom served about 10 million daily meeting participants. By April 2020 it was regularly serving more than 300 million a day. No capacity plan survives a thirtyfold demand jump in four months, and Zoom's CTO Brendon Ittelson was blunt about it: before the crisis Zoom by and large used its own data centers to handle traffic, and those data centers simply could not handle the surge.

Zoom's steady-state footprint was a set of colocation data centers (17 sites as of May 2020, per Data Center Frontier's reporting) acting as meeting zones close to users. What saved the platform was that this footprint was built to overflow into cloud regions. Zoom had run on AWS since 2011, and beginning in February 2020, AWS and Zoom engineering teams added tens of thousands of EC2 instances to Zoom's capacity, at times thousands of instances in a single day. Data Center Frontier reported Zoom provisioning 5,000 to 6,000 servers at a time from AWS as it expanded in-meeting video capacity. In November 2020 the two companies formalized this: AWS became Zoom's preferred cloud provider, running what CEO Eric Yuan described as the substantial majority of Zoom's cloud-based workloads.

The lesson before any diagram: real-time media capacity is horizontal. A meeting server fleet scales by adding identical machines behind a routing layer, which is exactly the shape of workload a cloud region can absorb overnight.

ZoomColo meeting zones overflow into elastic AWS regions
Colo meeting zones overflow into elastic AWS regionsMeeting participantsZoom colo data centers steady-state meeting zonesAWS CloudGlobal monitoringAWS Region (overflow zone)AWS Region (overflow zone)HTTPS: request nearest zoneassign steady-state zonesteer overflow to healthy zoneUDP media (steady state)UDP media (surge)UDP media (surge)+thousands of instances/dayload triggers scale-outZoom global backboneGlobalparticipants10M/day (Dec 2019) …Global cloudcontrollersteers meetings to …Colo MMR fleet17 colo meeting zon…Auto Scaling+thousands of insta…MMR fleet onEC2overflow meeting zo…MMR fleet onEC2identical zone arch…Auto Scalingscale out on loadMMR fleet onEC2overflow meeting zo…MMR fleet onEC2identical zone arch…CloudWatchfleet load metrics
Trace
Zoom's steady-state footprint was ~17 colocation meeting zones near users. Because every zone runs the identical MMR architecture, the global cloud controller can steer new meetings into AWS regions the moment their EC2 fleets are healthy. Beginning February 2020, Auto Scaling added thousands of EC2 instances a day to absorb the jump from ~10M to 300M+ daily participants.

The path a packet takes into a meeting

Zoom's own architecture documentation describes the traffic path in three moves. First, the client is steered by geolocation to the nearest available resources. This is a control-plane conversation: zone controllers in each meeting zone report status up to a global cloud controller, and the client is routed to the nearest data center and assigned to the least-loaded meeting server there.

Second, media flows. The meeting servers are multimedia routers (MMRs): they receive each participant's streams and switch them to the other participants rather than mixing them. Media rides UDP whenever the network allows, with seamless fallback to TCP/TLS on 443 in restrictive environments. UDP is the right transport for real-time media because a lost video packet is worthless by the time a retransmission arrives; the client instead publishes multiple stream layers and each receiver dynamically selects the appropriate layer while the app monitors bandwidth, packet loss, latency, and jitter in real time.

Third, the zones themselves are disposable and duplicated. Meeting zones are active-active, deployed with the same architecture in every data center, and traffic between zones can traverse Zoom's global backbone over dedicated links rather than the public internet. That symmetry is what made the 2020 surge tractable: an overflow meeting zone stood up in an AWS region is architecturally identical to a colo zone, so the global controller can steer new meetings into it the moment its MMR fleet is healthy. The client experience stays the same because the entry point stays near the user; only the zone answering grew elastic.

ZoomThe path a packet takes into a meeting: control plane, media plane, active-active zones
The path a packet takes into a meeting: control plane, media plane, active-active zonesMeeting participantsControl planeMeeting zone A (colo) nearest data centerAWS CloudMeeting zone B (AWS overflow) identical zone architecture1. HTTPS: which zone?report health/loadreport health/loadassign least-loaded MMR2. UDP media (layers)UDP mediamediaTCP/TLS 443 fallbackswitched streams (selected layer)switched streams3. Zoom backbone (inter-zone, active-active)Participant Apublishes multiple …Participant Bselects layer by ba…Participant Cbehind restrictive …Global cloudcontrollergeolocation steeringZone controllerAreports zone health…Meeting server(MMR)part of the zone fl…Meeting server(MMR)least-loaded target…RestrictivenetworkUDP blocked -> TCP/…Zone controllerBidentical active-ac…Meeting server(MMR)AWS overflow zone, …
Trace
Three moves. (1) Control plane: zone controllers report health and load up to the global cloud controller, which geolocation-steers the client to the nearest data center and its least-loaded meeting server. (2) Media plane: streams ride UDP to a multimedia router (MMR) that switches them to other participants rather than mixing, falling back to TCP/TLS 443 in restrictive networks. (3) Zones are active-active and identical, linked over Zoom's private backbone.

What this teaches for the exam

Task 1.1 is about edge services for global performance, and Zoom's design is the canonical argument for regional entry points. Real-time UDP media cannot be cached, so a CDN like CloudFront (an HTTP cache) is the wrong tool; the pattern that fits is AWS Global Accelerator, which gives users a nearby anycast entry point, supports UDP as well as TCP, and carries traffic over the AWS backbone instead of the public internet. That is the managed equivalent of Zoom's geolocation steering plus private backbone. When a question pairs 'real-time', 'UDP', or 'gaming/voice/video' with 'global users', think Global Accelerator in front of Network Load Balancers; when it pairs 'static or cacheable HTTP' with global users, think CloudFront.

Task 3.3 is optimizing networks for performance, reliability, and cost, and three Zoom moves map directly. Directing each user to the nearest healthy region is what Route 53 latency-based routing with health checks does; least-loaded assignment inside a region is a load balancing and traffic distribution pattern. Sending each receiver only the stream layer it can handle is a bandwidth-reduction technique, the same category as choosing multicast over repeated unicast or fronting content with a cache. And the surge itself is the throughput story: media servers are unicast packet pumps, so fleet capacity is network throughput times instance count, which on AWS means ENA-based instances sized for packets per second and scaled horizontally, exactly what adding thousands of EC2 instances a day was doing.

One honest boundary: Zoom has not published which AWS networking primitives it uses inside its regions, so do not quote Zoom as evidence for any specific service choice. What the public record supports is the shape: regional entry close to the user, UDP end to end, health-and-load-aware steering, identical zones everywhere, and a cloud region as instant overflow capacity.

ZoomThe managed AWS equivalent: Global Accelerator, NLB, and Route 53 latency steering
The managed AWS equivalent: Global Accelerator, NLB, and Route 53 latency steeringInternet clientsAWS CloudGlobal edge servicesus-east-1 (nearest healthy)eu-west-1 (failover)Media VPC 10.0.0.0/16Media VPC 10.1.0.0/16Public subnetsPrivate subnetsPublic subnetsPrivate subnets1. DNS: nearest healthy region2. UDP to nearest anycast edgeover AWS backboneover AWS backbonehealth checkhealth checkleast-loaded MMRleast-loaded MMRscale out (pps)scale out (pps)Global usersreal-time UDP videoRoute 53latency-based routi…GlobalAcceleratoranycast entry, UDP …Network LoadBalancerleast-loaded distri…Auto Scalingscale out on packet…MMR on ENAinstancesized for packets p…MMR on ENAinstancehorizontal fleetNetwork LoadBalancerleast-loaded distri…Auto Scalingscale out on packet…MMR on ENAinstancesized for packets p…MMR on ENAinstancehorizontal fleet
Trace
The public record supports the shape, not Zoom's exact services. Real-time UDP media cannot be cached, so CloudFront is wrong; the fit is Global Accelerator giving a nearby anycast entry that carries UDP over the AWS backbone to regional Network Load Balancers. Route 53 latency-based routing with health checks steers each user to the nearest healthy region, NLBs do least-loaded distribution, and MMRs run on ENA-based EC2 sized for packets per second and scaled horizontally.

More diagrams

Zoom meeting traffic path: geolocation entry, MMR fleets, and AWS surge capacity
Zoom meeting traffic path: geolocation entry, MMR fleets, and AWS surge capacityMeeting participantsMeeting zonesZoom colo data center nearest meeting zoneAWS CloudAWS Region surge meeting zoneHTTPS: assign zoneUDP mediaUDP media (surge)Zoom backboneParticipant AUDP media, TCP 443 …Participant BUDP mediaGlobal cloudcontrollergeolocation steeringMMR fleetmultimedia routersMMR fleet onEC2thousands of instan…
Trace
Clients ask the control plane for the nearest meeting zone over HTTPS, then send media over UDP to a multimedia router (MMR) in that zone. During the 2020 surge, Zoom's colo zones ran out of headroom and overflow meeting zones were stood up on EC2, with thousands of instances added in a day.

Sources

Practice what you just read

6 questions from this architecture

Loading…