# Postmortem: Routing Instability and Partial Network Outage

**Incident date:** July 18, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Authors:** Network Operations and Infrastructure Engineering

## Summary

On July 18, a configuration change caused repeated route withdrawals and reconvergence across the community mesh network. Customers connected through seven neighborhood relay sites experienced intermittent or complete loss of internet access for 2 hours and 47 minutes.

The change enabled a new route-export policy on two recently upgraded gateway routers. Due to an incorrect prefix filter, each gateway advertised routes learned from the other back into the mesh. This created a routing feedback loop that caused frequent path changes, overloaded several lower-powered relay nodes, and made customer traffic unreliable.

Engineers stabilized the network by disabling route redistribution on the affected gateways, restarting routing processes on overloaded relays, and restoring the previous configuration.

## Customer Impact

Approximately 1,840 of 4,900 active subscribers were affected.

- 620 customers experienced a complete loss of connectivity for between 38 and 112 minutes.
- Approximately 1,220 customers experienced intermittent connectivity, high packet loss, or latency above 1,000 ms.
- Seven neighborhood relay sites were affected.
- Voice-over-IP calls, video meetings, and online gaming were particularly disrupted.
- The customer portal remained available from outside the mesh, but many affected customers could not reach it.
- Support received 286 calls, 94 emails, and 173 text messages during the incident.

No customer data was lost, and there was no evidence of unauthorized access.

## Detection

The incident was first detected through customer reports. Automated monitoring generated alerts for elevated gateway CPU usage and relay-node packet loss, but these alerts were initially classified as separate site-level problems rather than a network-wide routing incident.

The first correlated routing alert was generated 31 minutes after customer impact began. It was delayed because the route-change threshold had been tuned for short reconvergence events common in a wireless mesh.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 18:42 | An engineer deploys a scheduled routing-policy update to gateways `gw-east-2` and `gw-central-1`. Pre-deployment validation passes. |
| 18:45 | Both gateways begin re-advertising routes learned through the mesh. Route updates increase sharply. |
| 18:47 | Customer impact begins. Relay nodes in Riverside and East Junction show increased CPU usage and packet loss. |
| 18:51 | Monitoring alerts on high CPU usage at three relay sites. The alerts are assigned to the wireless operations queue. |
| 18:56 | The first customer reports intermittent connectivity. |
| 19:05 | Support identifies reports from multiple neighborhoods and escalates to the on-call network engineer. |
| 19:13 | The on-call engineer suspects wireless interference because several affected links use the same 5 GHz channel. Channel changes are attempted at two sites without improvement. |
| 19:18 | Two low-powered relay nodes become unresponsive under routing-process load. Customers behind those relays lose connectivity completely. |
| 19:29 | Monitoring generates a route-churn alert after the sustained threshold is exceeded. |
| 19:34 | The incident is declared SEV-1. A response channel and conference bridge are opened. |
| 19:41 | Engineers identify rapidly changing next hops for customer and infrastructure prefixes. |
| 19:52 | Comparison with the previous gateway configuration reveals a missing inbound-prefix constraint in the new route-export policy. |
| 20:01 | Engineers disable route redistribution on `gw-central-1`. Route churn decreases but continues through `gw-east-2`. |
| 20:08 | Route redistribution is disabled on `gw-east-2`. The control plane begins to stabilize. |
| 20:17 | Several relay nodes remain overloaded and require routing-process restarts. |
| 20:31 | Core routes are stable. Most customers regain connectivity, though two relay sites remain isolated. |
| 20:46 | Field volunteers power-cycle one unresponsive relay node at the Riverside community center. |
| 21:06 | The final isolated relay reconnects after its routing process is restarted remotely through the management radio. |
| 21:18 | Packet loss and latency return to normal across all monitored paths. |
| 21:29 | The incident is resolved after 15 minutes of stable routing and successful customer-path tests. |

## Root Cause

The outage was caused by an incorrect route-export policy deployed to two internet gateway routers.

The intended policy allowed each gateway to advertise locally originated customer prefixes and a default route into the mesh. Instead, the policy also exported routes that the gateway had learned from other mesh nodes. Because both gateways received and re-advertised these routes, they repeatedly presented alternative paths to the same destinations.

The mesh routing protocol responded by recalculating paths and propagating updates. This behavior was technically valid, but the volume and frequency of updates exceeded the processing capacity of several relay nodes. As those nodes fell behind, route adjacencies expired and reformed, producing additional updates and amplifying the instability.

The configuration error was not rejected because it was syntactically valid and passed existing validation checks.

## Contributing Factors

- The staging environment included only one internet gateway and could not reproduce feedback between multiple gateways.
- Configuration validation checked syntax and route availability but did not verify route origin or detect re-advertisement.
- The deployment updated both gateways within the same maintenance step, eliminating a stable comparison point.
- Several relay nodes use older single-board computers with limited CPU and memory.
- Route-churn alerts tolerated sustained instability for too long because brief topology changes are common in the mesh.
- Initial alerts were grouped by relay site rather than correlated across the routing domain.
- The rollback procedure restored configuration files but did not automatically clear previously learned routes.
- The first response focused on radio interference, delaying investigation of the routing control plane.
- The incident occurred after normal volunteer staffing hours, increasing response and on-site access times.

## Resolution and Recovery

Engineers stopped the feedback loop by disabling route redistribution on both affected gateways and restoring the previous export policy. Routing processes were then restarted on relay nodes that had not recovered automatically.

One relay node at the Riverside community center was no longer reachable through either the production network or its management radio. A local volunteer accessed the site and power-cycled the device.

After all routing adjacencies stabilized, engineers verified connectivity from each neighborhood relay to both upstream providers and tested representative customer paths. The incident remained open for an additional 15-minute observation period before being marked resolved.

## What Went Well

- Support recognized that reports spanned multiple neighborhoods and escalated the issue.
- Out-of-band management radios allowed remote recovery of all but one relay node.
- Configuration versioning made it possible to identify the relevant change quickly.
- Local volunteers were available to access the Riverside site.
- No data-plane equipment or customer equipment required replacement.
- Upstream transit providers remained available throughout the incident.

## What Went Poorly

- Automated detection did not identify the incident before customers reported it.
- The deployment process allowed simultaneous changes to both gateway routers.
- The test environment did not represent the production gateway topology.
- Existing validation did not check route provenance or export-policy behavior.
- The incident runbook emphasized radio and power failures but provided little guidance for routing instability.
- Customer communication began 52 minutes after impact started.
- Support staff lacked a current map linking customer areas to relay and gateway dependencies.

## Corrective and Preventive Actions

| Action | Owner | Priority | Target Date | Status |
|---|---|---:|---|---|
| Add automated tests that reject export of mesh-learned routes to other mesh peers | Network Engineering | P0 | July 25, 2026 | Complete |
| Require canary deployment to one gateway followed by a 15-minute observation period | Network Operations | P0 | July 25, 2026 | Complete |
| Add maximum-prefix and route-origin safeguards to every gateway peer | Network Engineering | P0 | August 1, 2026 | In progress |
| Build a multi-gateway routing lab that mirrors production policy behavior | Infrastructure Engineering | P1 | August 15, 2026 | In progress |
| Reduce the route-churn alert window from 30 minutes to 5 minutes, with suppression for planned maintenance | Observability | P1 | August 5, 2026 | Planned |
| Correlate CPU, adjacency, and packet-loss alerts across the routing domain | Observability | P1 | August 19, 2026 | Planned |
| Update the SEV-1 runbook with routing-loop diagnostics and rollback steps | Network Operations | P1 | August 8, 2026 | Planned |
| Automate clearing of affected routing sessions during rollback | Network Engineering | P1 | August 12, 2026 | Planned |
| Upgrade the five relay nodes most vulnerable to control-plane overload | Community Infrastructure | P1 | September 15, 2026 | Scheduled |
| Add customer-status notifications through SMS and an externally hosted status page | Customer Operations | P2 | August 30, 2026 | Planned |
| Publish and maintain a relay-to-neighborhood dependency map for support staff | Customer Operations | P2 | August 12, 2026 | Planned |
| Conduct a routing-failure exercise with staff and volunteer operators | Incident Management | P2 | September 5, 2026 | Planned |

## Lessons Learned

A mesh network can tolerate individual link and node failures, but that resilience does not protect it from incorrect routing information distributed through the control plane. Gateway policy changes must be treated as high-risk changes even when they are small and syntactically valid.

Future gateway deployments will be tested against a multi-gateway topology, introduced through a canary, and monitored for route provenance and churn before being rolled out across the network.