When a subsea cable was cut off the coast of West Africa last spring, traffic for thousands of applications had to find a new way home within seconds. On our network it took 2.4 seconds, and most users never noticed. Here is how Anycast and a little custom BGP make that possible.
Anycast in one paragraph
With Anycast, the same IP prefix is announced from every one of our edge locations at once. Internet routers choose the path that looks shortest to them, so each user lands at a nearby point of presence without any DNS tricks. If one location disappears, its announcement is withdrawn and the next nearest takes over automatically.
Why default BGP is not enough
Plain BGP only knows about paths, not about how those paths are performing. A route can be technically available yet badly congested, and the protocol will keep sending traffic down it until someone intervenes. During an outage, upstream providers often reroute onto the same few links, which is exactly when congestion peaks.
We needed routing decisions that react to measured latency and loss, not just reachability.
Reachability tells you a road exists. Measurement tells you whether it is jammed. Good routing needs both.
Our peering engine
Every edge location continuously probes thousands of destinations across each of its transit and peering links. A controller collects those measurements every second and adjusts which prefixes each location announces, and with which path attributes, to steer traffic away from links that are degrading.
Changes are rate limited and staged so that the network never oscillates. A location first prepends its path to make itself less attractive, then withdraws only if conditions keep getting worse.
probe: interval: 1s targets: top-5000-asns steer: latency_threshold_ms: 35 loss_threshold_pct: 0.5 actions: [prepend, withdraw] max_changes_per_minute: 6
Anatomy of a cable cut
When the cable failed, loss on two of our Lagos transit links jumped from almost zero to 38 per cent within half a second. The controller saw the change on its next cycle, prepended those links, and shifted West African traffic onto our private fibre to Lisbon and a regional peering exchange.
- 0.0 s: loss detected on two transit links
- 1.1 s: path prepends announced from the affected location
- 2.4 s: 97 per cent of traffic flowing over alternative paths
- 41 min: original links restored and traffic gradually returned
- Anycast gives automatic failover between locations without DNS changes.
- BGP alone reacts to failures, not to degradation.
- Measured latency and loss should drive routing announcements.
- Staged, rate-limited changes keep a large network stable.