# Postmortem: Mesh Routing Instability and Partial Internet Outage **Incident date:** 2026-07-12 **Duration:** 2 hours 47 minutes **Severity:** SEV-1 **Status:** Resolved **Authors:** Network Operations and Infrastructure Engineering ## Summary On July 12, a configuration rollout caused routing instability across the community mesh network. Approximately 68% of subscribers experienced intermittent connectivity, high latency, or complete loss of internet access for up to 2 hours and 47 minutes. The rollout reduced the routing protocol's neighbor timeout to improve recovery from failed wireless links. On several busy relay nodes, normal packet loss and CPU contention caused healthy neighbors to exceed the new timeout. Nodes repeatedly withdrew and relearned routes, producing route flapping that spread through the mesh and overloaded the two primary internet gateways. Service was restored by reverting the timeout change, restarting routing processes on the most affected relays, and temporarily limiting route updates at the gateways. ## Customer Impact - Approximately 1,840 of 2,710 active subscribers were affected. - About 920 subscribers experienced at least one period of complete disconnection. - Other affected subscribers saw packet loss between 15% and 60%, latency above 2 seconds, and frequent connection resets. - Voice and video calls were generally unusable in the central and northwest neighborhoods. - Locally hosted community services remained reachable within some mesh segments, but route instability made access intermittent. - The customer portal and status page remained online because they are hosted outside the mesh. - No customer data was lost or exposed. ## Detection The incident was first detected through customer reports in the community support channel. Automated monitoring alerted three minutes later on elevated gateway packet loss, but it did not identify mesh-wide route churn as the underlying condition. Our monitoring sampled routing-table size every five minutes. This was too coarse to capture routes being rapidly withdrawn and restored, so dashboards initially suggested that route counts were within their normal range. ## Timeline All times are in Eastern Daylight Time. - **18:02** — Automation begins deploying the revised neighbor timeout to relay nodes. - **18:09** — Deployment completes on 46 of 49 targeted nodes. Three offline nodes are skipped. - **18:14** — The first affected relay begins repeatedly losing and reestablishing neighbors. - **18:19** — Route churn spreads to adjacent mesh segments. - **18:23** — Customers begin reporting intermittent connectivity. - **18:26** — Monitoring alerts on elevated packet loss at both primary gateways. - **18:31** — The on-call engineer acknowledges the alert and begins investigating upstream connectivity. - **18:38** — Upstream transit providers confirm that their links are healthy. - **18:44** — Gateway CPU utilization reaches 96% as routing updates accumulate. - **18:51** — The incident is declared SEV-1. Network Operations and Infrastructure Engineering join the response. - **19:03** — Engineers identify unusually frequent neighbor-state changes on central relay nodes. - **19:12** — The team correlates the instability with the timeout configuration rollout. - **19:18** — A rollback is initiated through the deployment system. - **19:27** — The rollback stalls on 11 nodes with saturated management links. - **19:34** — Engineers begin applying the previous configuration through alternate paths. - **19:47** — Route-update rate limiting is enabled at both gateways, reducing CPU utilization. - **20:02** — The highest-traffic relays are restarted in a controlled sequence. - **20:18** — Neighbor sessions stabilize across the central mesh. - **20:31** — Packet loss returns below 3% for most subscribers. - **20:42** — The final isolated northwest segment reconnects after its relay configuration is reverted. - **20:49** — Service metrics return to normal. The incident remains under observation. - **21:16** — The incident is closed after 27 minutes of stable routing and gateway performance. ## Root Cause The immediate cause was a reduction of the routing protocol's neighbor timeout from 12 seconds to 4 seconds. The new value had been tested in a lab and on three low-traffic field nodes. Those environments did not reproduce the combination of wireless packet loss, routing-process scheduling delays, and CPU contention present on high-degree relay nodes during peak usage. Under those conditions, some valid neighbor announcements arrived after the four-second timeout. Nodes treated healthy links as failed, withdrew routes using those links, and then restored them when the next announcement arrived. Each transition generated additional routing updates. Because several central relays connect many mesh segments, their route flapping propagated widely. The resulting update volume increased CPU and queue utilization on neighboring nodes and internet gateways. This delayed routing traffic further, causing more neighbor timeouts and creating a positive feedback loop. ## Contributing Factors - The deployment changed all reachable relays within seven minutes instead of using staged neighborhood cohorts. - The canary nodes had lower traffic and fewer neighbors than the relays most exposed to the failure mode. - The configuration validator checked syntax and allowed ranges but did not enforce the operational minimum timeout documented by the routing team. - Monitoring tracked routing-table size but not route withdrawals, neighbor flaps, or update frequency. - Management traffic shared queues with subscriber traffic on several older relays, making rollback access unreliable during congestion. - The runbook initially directed responders to investigate upstream transit before checking mesh routing stability. - Three nodes missed the rollout because they were offline, creating inconsistent timeout behavior when they later rejoined. ## Resolution and Recovery We reverted the neighbor timeout to 12 seconds across the network. Where normal management access was unavailable, engineers used alternate mesh paths and direct site access. We also rate-limited routing updates at the internet gateways and restarted routing processes on six heavily affected relays in a controlled order. These measures reduced processing pressure long enough for neighbor sessions and routing tables to converge. The three nodes that missed the original deployment were inspected before being returned to service and confirmed to have the restored configuration. ## What Went Well - Community members reported symptoms quickly and included useful neighborhood and node information. - Upstream providers responded promptly and helped rule out transit failure. - Configuration history allowed responders to identify and revert the relevant change without reconstructing the previous state. - Alternate management paths remained available for most critical relays. - No hardware replacement was required. ## What Went Poorly - The rollout reached most of the network before abnormal behavior was detected. - The selected canaries did not represent high-traffic, high-degree relay nodes. - Existing dashboards obscured rapid routing changes by showing five-minute aggregates. - Rollback automation was not designed for a congested or unstable management plane. - The first customer update was posted 38 minutes after reports began and underestimated the affected area. ## Corrective and Preventive Actions | Action | Owner | Priority | Due date | Status | |---|---|---:|---:|---| | Restore and lock the neighbor timeout at 12 seconds pending further testing | Network Operations | P0 | 2026-07-12 | Complete | | Add validation preventing timeouts below the documented operational minimum | Infrastructure Engineering | P0 | 2026-07-19 | In progress | | Implement staged rollouts by neighborhood with automatic pause criteria | Infrastructure Engineering | P0 | 2026-07-31 | In progress | | Select canaries that include high-degree, high-traffic, and older relay hardware | Network Engineering | P1 | 2026-07-24 | Planned | | Alert on neighbor flaps, route withdrawals, and routing-update frequency | Observability | P0 | 2026-07-26 | In progress | | Add one-minute and ten-second routing metrics to incident dashboards | Observability | P1 | 2026-08-02 | Planned | | Separate or prioritize management traffic on legacy relays | Network Engineering | P1 | 2026-08-30 | Planned | | Make rollback packages available locally on relay nodes | Infrastructure Engineering | P1 | 2026-08-09 | Planned | | Test rollback procedures under simulated congestion and route churn | Reliability Working Group | P1 | 2026-08-16 | Planned | | Update the outage runbook to check mesh convergence before upstream transit | Network Operations | P1 | 2026-07-22 | Planned | | Automatically reconcile nodes that miss a configuration deployment | Infrastructure Engineering | P2 | 2026-08-23 | Planned | | Establish a 20-minute target for the first customer-facing incident update | Community Support | P1 | 2026-07-21 | Planned | ## Lessons Learned Routing timers in a wireless mesh are coupled to real-world link quality, node load, and topology. A value that improves failure detection in a controlled environment can reduce reliability when applied to busy relay nodes. Future routing changes will be evaluated against representative peak-load conditions and deployed in small, observable cohorts. Rollouts will automatically stop when neighbor flaps, route withdrawals, packet loss, or gateway CPU utilization exceed defined thresholds.