# Postmortem: Gateway Routing Failure and Partial Network Outage **Incident date:** July 24, 2026 **Duration:** 2 hours 47 minutes **Severity:** SEV-1 **Status:** Resolved **Authors:** Network Operations and Community Infrastructure Teams ## Summary On July 24, a routing configuration change caused two core gateways to advertise conflicting default routes into the community mesh. Many nodes selected an unavailable transit path, resulting in intermittent or complete loss of internet access across several neighborhoods. The outage began at 18:12 EDT and ended at 20:59 EDT after operators withdrew the incorrect route advertisements and restarted routing services on affected relay nodes. Approximately 61% of active subscribers experienced disruption. ## Customer Impact An estimated 1,840 households were affected: - 1,120 households lost internet connectivity entirely. - 720 households experienced intermittent connectivity, high latency, or packet loss. - Local mesh services remained reachable in most areas. - Voice-over-IP calling and payment terminals using our connection were disrupted. - Network status updates were delayed because the public status page depended on the affected infrastructure. Customers experienced impact for between 34 minutes and 2 hours 47 minutes, depending on their node’s route selection and convergence time. ## Timeline All times are in Eastern Daylight Time. - **17:45** — An engineer begins scheduled maintenance to migrate Gateway East to a new upstream provider. - **18:07** — A routing configuration is generated from an outdated template containing a default-route announcement intended only for lab gateways. - **18:12** — The configuration is deployed to Gateway East. It begins advertising a default route before its new upstream link is operational. - **18:14** — Nearby relay nodes begin selecting Gateway East as their preferred internet path. - **18:19** — Automated monitoring detects elevated packet loss but classifies it as a neighborhood-level wireless issue. - **18:26** — The support queue receives the first reports of complete connectivity loss. - **18:33** — Gateway West also begins advertising a default route with the same preference after an automated configuration synchronization. - **18:41** — The on-call operator acknowledges the alert and begins investigating wireless backhaul links. - **18:57** — Customer reports indicate that the incident spans multiple neighborhoods. - **19:08** — The incident is escalated to SEV-1, and the network operations lead joins. - **19:21** — Engineers identify rapidly changing default routes on several relay nodes but initially suspect a routing-daemon defect. - **19:39** — Configuration differences reveal the unintended default-route advertisement on Gateway East. - **19:46** — The team disables configuration synchronization to prevent further propagation. - **19:52** — The incorrect advertisement is withdrawn from Gateway East. - **20:03** — Most northern neighborhoods recover, but several relay nodes continue using stale routes through Gateway West. - **20:17** — The same configuration error is confirmed on Gateway West. - **20:24** — The incorrect advertisement is withdrawn from Gateway West. - **20:31** — Routing services are restarted in batches on relay nodes that did not reconverge automatically. - **20:47** — Connectivity is restored for more than 95% of affected customers. - **20:59** — Packet loss and route-selection metrics return to normal. The incident is declared resolved. - **22:15** — Engineers complete a review of all gateway configurations and confirm that no other nodes contain the invalid policy. ## Root Cause The immediate cause was an invalid routing policy deployed to Gateway East. The policy advertised a default route into the mesh even though the gateway did not yet have a working upstream connection. The configuration was generated from an outdated template copied from the lab environment. That template did not include the production safeguard requiring upstream reachability before a gateway could originate a default route. An automated synchronization process then copied the relevant policy to Gateway West. Because both gateways advertised routes with equal preference, traffic shifted unpredictably between them. Some relay nodes also retained stale routes after the advertisements were withdrawn, extending the recovery period. ## Contributing Factors - Gateway configuration templates were stored separately from the reviewed production configuration repository. - The deployment system validated syntax but did not simulate routing behavior. - Default-route advertisements were not gated on upstream health. - Configuration synchronization included routing-policy sections that should have required separate approval. - Monitoring focused on gateway availability rather than whether customer traffic successfully reached the public internet. - The status page was hosted behind the same gateway infrastructure affected by the incident. - The initial alert resembled common wireless-link degradation and did not indicate the geographic scope of the failure. - Some relay nodes ran an older routing-daemon version with slower stale-route expiration. ## Resolution and Recovery Operators stopped configuration synchronization, removed the incorrect default-route policy from both gateways, and verified that only gateways with healthy upstream links were advertising internet connectivity. Most nodes recovered through normal routing convergence. Relay nodes that retained stale routes were restarted in controlled batches. Engineers then confirmed recovery using external probes, customer-premises telemetry, and test traffic from each affected neighborhood. ## What Went Well - Community volunteers provided reports from multiple neighborhoods, helping establish the incident’s scope. - The routing configuration could be rolled back without physical access to gateway sites. - Local mesh connectivity remained available, allowing field operators to reach several affected nodes. - Once the conflicting advertisements were identified, the mitigation was low risk and restored most service quickly. ## What Went Poorly - The change was deployed without a second-person review. - Monitoring detected packet loss but pointed responders toward the wrong failure domain. - The configuration synchronization process expanded the incident from one gateway to two. - Customers had no reliable source of updates while the status page was unavailable. - Node software inconsistencies made recovery slower and less predictable. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---| | Move all gateway templates into the reviewed production configuration repository | Network Engineering | P0 | August 7, 2026 | | Require peer approval for changes affecting route origination or preference | Infrastructure Lead | P0 | August 7, 2026 | | Gate default-route advertisements on verified upstream reachability | Network Engineering | P0 | August 14, 2026 | | Add pre-deployment tests that simulate route selection across representative mesh topologies | Platform Engineering | P1 | August 28, 2026 | | Restrict automated synchronization from modifying routing policies | Platform Engineering | P0 | August 7, 2026 | | Add external connectivity probes for every gateway and neighborhood relay cluster | Observability Team | P1 | August 21, 2026 | | Alert on unexpected default-route originators and rapid next-hop changes | Observability Team | P1 | August 14, 2026 | | Host the public status page outside the production mesh infrastructure | Operations | P1 | August 21, 2026 | | Upgrade all relay nodes to the current routing-daemon release | Community Infrastructure | P2 | September 30, 2026 | | Document a routing-loop and blackhole response procedure | Network Operations | P1 | August 14, 2026 | | Conduct a failure exercise involving loss of upstream connectivity at a core gateway | Incident Management | P2 | September 15, 2026 | ## Lessons Learned In a mesh network, a single incorrect route advertisement can affect nodes far beyond the gateway where the change originated. Syntax validation and gateway-level health checks were insufficient because they did not verify the path customers would actually use. Future routing changes will be tested against representative topology data, reviewed by a second engineer, and deployed with automated safeguards that prevent gateways without working upstream connectivity from advertising themselves as internet exits.