# Postmortem: Mesh Routing Instability and Partial Internet Outage **Incident date:** July 18, 2026 **Duration:** 2 hours 47 minutes **Severity:** SEV-1 **Status:** Resolved **Authors:** Network Operations and Infrastructure Engineering ## Summary On July 18, a configuration change caused unstable route selection across the community mesh network. Nodes repeatedly switched between two gateway paths, producing routing loops, packet loss, and intermittent internet access. Approximately 1,840 subscribers experienced degraded service, including 690 who were effectively offline for most of the incident. Local traffic within individual neighborhoods generally continued to work, but services requiring an upstream internet connection were unreliable. Engineers stabilized the network by withdrawing one gateway, reverting the configuration, and restarting routing processes on affected relay nodes. No customer data was lost. ## Customer Impact From 18:14 to 21:01 local time: - 1,840 of 2,730 active subscribers experienced elevated latency or packet loss. - Approximately 690 subscribers had no usable internet connection. - Voice-over-IP calls frequently dropped or failed to connect. - Video meetings and streaming services stalled or disconnected. - The customer portal and status page were intermittently unreachable because they depended on the affected network. - Neighborhood-local services remained available where devices shared an unaffected mesh segment. The incident disproportionately affected customers connected through the North Hill and Riverside relay clusters. Some customers experienced an additional 10–15 minutes of disruption while their rooftop nodes re-established routes after recovery. ## Detection Automated monitoring detected elevated packet loss at 18:16, two minutes after customer impact began. The alert was initially classified as a single upstream-provider problem because both internet gateways remained reachable from the monitoring server. The on-call engineer identified widespread routing instability at 18:29 after comparing route advertisements from multiple relay nodes. Customer reports began arriving through the support phone line at 18:20. ## Timeline All times are local. - **17:52** — An engineer begins a planned change to prefer the newly upgraded Riverside gateway during peak traffic. - **18:08** — The updated route-policy template is deployed to 14 backbone relay nodes. - **18:14** — Relay nodes begin alternating between the Riverside and Central Library gateways. Packet loss rises sharply. - **18:16** — Monitoring alerts on packet loss and latency. The alert is automatically associated with the Riverside upstream circuit. - **18:20** — Support receives the first customer calls reporting intermittent connectivity. - **18:24** — The on-call engineer confirms that both upstream circuits are operational. - **18:29** — Route logs show frequent gateway changes and inconsistent path metrics between relay nodes. - **18:37** — The team pauses all scheduled network changes and begins rolling back the route-policy template. - **18:45** — Rollback completes, but instability continues because several nodes retain routes learned during the oscillation. - **18:53** — The incident is escalated to SEV-1. Additional network engineers join. - **19:07** — Engineers identify a routing loop between the North Hill and Riverside backbone clusters. - **19:18** — The Riverside gateway is temporarily withdrawn from the mesh. - **19:24** — Packet loss decreases, but six relay nodes continue advertising stale paths. - **19:41** — Routing processes are restarted sequentially on affected relay nodes. - **20:06** — Most customers regain stable connectivity. - **20:19** — Engineers begin restoring the Riverside gateway with its previous policy. - **20:43** — Both gateways are operating normally, and routing tables are stable. - **21:01** — Customer-impact metrics return to baseline. The incident is resolved. - **22:10** — The public status page is updated with a preliminary explanation. ## Root Cause The outage was caused by an incorrect route-policy template deployed during a gateway preference change. The mesh uses dynamic routing metrics to choose between two internet gateways. The new template lowered the Riverside gateway's advertised cost but omitted a required policy that adds cost when a route is re-advertised between backbone clusters. Without that safeguard, North Hill preferred a Riverside route learned through Central Library, while Central Library simultaneously preferred a Riverside route learned back through North Hill. Traffic entered a routing loop until route timers expired. Because wireless link quality fluctuates naturally, the preferred paths changed repeatedly, causing widespread route oscillation. The deployment system validated the template's syntax but did not simulate route selection across multiple clusters. The change therefore passed validation even though it created an invalid network-wide routing state. ## Contributing Factors - The change was deployed to all backbone relays at once rather than to a small canary group. - The route-policy test environment represented individual nodes but not a multi-cluster mesh topology. - Monitoring checked gateway reachability but did not alert on route churn or forwarding loops. - The rollback restored configuration files but did not clear routes already learned by running routing processes. - The customer status page was hosted behind the same network gateways affected by the incident. - The maintenance plan did not define an explicit threshold for aborting the rollout. - The change was reviewed by one engineer, although it affected the network-wide routing policy. ## Resolution and Recovery Engineers first reverted the configuration change, then temporarily withdrew the Riverside gateway to force all external traffic through Central Library. They restarted routing processes on relay nodes that retained stale paths and confirmed that forwarding tables converged. After the network remained stable for 20 minutes, the Riverside gateway was restored using the previous route policy. Engineers monitored route changes, packet loss, latency, and gateway traffic until all metrics returned to normal. ## What Went Well - Packet-loss monitoring detected the incident within two minutes. - Community volunteers quickly confirmed that the issue affected multiple neighborhoods. - Engineers had current configuration backups and completed the rollback without data loss. - The second gateway had enough spare capacity to carry traffic during recovery. - Sequential routing-process restarts avoided another network-wide reconvergence event. ## What Went Poorly - The initial alert classification delayed investigation of the routing layer. - Existing validation could not detect a policy error spanning multiple nodes. - The all-at-once deployment expanded the blast radius. - Rolling back the configuration did not fully restore the previous runtime state. - Customers could not reliably reach the primary status page during the outage. - Internal documentation did not clearly describe the purpose of the omitted route-cost rule. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---:| | Add a multi-cluster mesh topology to the routing test environment | Network Engineering | P0 | July 31 | | Implement route-loop and route-churn checks in pre-deployment validation | Infrastructure Engineering | P0 | August 7 | | Require canary deployment to one relay per cluster before network-wide routing changes | Network Operations | P0 | Completed | | Add automated rollback when route changes exceed the defined stability threshold | Infrastructure Engineering | P1 | August 21 | | Update rollback tooling to clear or refresh affected runtime routing state | Network Engineering | P1 | August 14 | | Alert on abnormal route-update frequency and next-hop oscillation | Observability Team | P1 | August 14 | | Host a secondary status page outside the community mesh network | Operations | P1 | August 7 | | Require two-person review for gateway and backbone routing-policy changes | Engineering Manager | P1 | Completed | | Document all routing-policy invariants and the reason for each safeguard | Network Engineering | P2 | August 28 | | Run a quarterly gateway-failover and route-convergence exercise | Network Operations | P2 | September 15 | ## Lessons Learned A configuration can be syntactically valid and locally reasonable while still being unsafe for a distributed network. Future routing changes will be evaluated as changes to the full mesh topology, not as isolated node updates. We also need recovery procedures that restore runtime state, not only configuration files. Finally, customer communications must remain available when the network itself is impaired.