# Postmortem: East Ridge Mesh Outage

**Date:** 2026-06-18  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Services affected:** Residential internet access, community Wi-Fi hotspots, monitoring telemetry

## Summary

On June 18, the East Ridge mesh experienced a widespread outage after a routing configuration change was deployed to two rooftop relay nodes during scheduled maintenance. The change caused unstable Babel route advertisements between the East Ridge and Hilltop sectors. Several mesh nodes repeatedly selected a congested backup path over the primary fiber-fed gateway, creating a routing loop and saturating the 5 GHz backhaul.

The outage was resolved by rolling back the Babel configuration on the affected relay nodes and manually clearing stale routes across the East Ridge sector.

## Customer Impact

- Approximately 312 subscriber households were affected.
- 221 households experienced complete loss of connectivity.
- 91 households experienced severe packet loss, high latency, or intermittent service.
- Three public community Wi-Fi hotspots were offline.
- VoIP customers in the affected area saw failed or dropped calls.
- Monitoring was partially degraded because several node exporters became unreachable through the mesh.

No customer data was lost or exposed.

## Timeline

All times are local.

| Time | Event |
|---|---|
| 01:00 | Scheduled maintenance window begins. |
| 01:07 | Updated Babel routing configuration deployed to `er-relay-03` and `er-relay-04`. |
| 01:10 | First packet loss alerts fire for East Ridge subscriber CPEs. |
| 01:13 | On-call engineer acknowledges alerts and begins checking gateway health. |
| 01:18 | Hilltop gateway confirmed healthy; fiber uplink operating normally. |
| 01:24 | Mesh graph shows repeated next-hop changes between East Ridge and Hilltop sectors. |
| 01:31 | Customer reports begin arriving via SMS support line. |
| 01:39 | On-call identifies abnormal route advertisements from `er-relay-03`. |
| 01:45 | Babel configuration rollback started on `er-relay-03` and `er-relay-04`. |
| 01:53 | Route flapping decreases, but several nodes continue using stale backup paths. |
| 02:06 | Engineers manually restart Babel on six affected relay nodes. |
| 02:18 | Primary gateway route becomes stable for most East Ridge nodes. |
| 02:31 | Packet loss returns to normal baseline. |
| 02:42 | Incident declared resolved. |
| 03:15 | Post-incident monitoring confirms stable routing and normal backhaul utilization. |

## Root Cause

The outage was caused by an incorrect Babel configuration template deployed to two East Ridge relay nodes. The template changed the route preference for backup wireless paths without including the intended interface-specific metric penalties.

As a result, `er-relay-03` and `er-relay-04` advertised backup mesh paths as equally preferred to the primary gateway path. Nearby nodes began switching between routes based on small changes in link quality. This created rapid route churn and, in some cases, transient routing loops.

The routing instability saturated one Hilltop-to-East Ridge backhaul link, which made the backup path appear worse, causing additional route changes. The mesh did not fully recover until stale routes were cleared from downstream relays.

## Contributing Factors

- The configuration change was tested on a lab node but not on a multi-hop sector topology.
- The rollout affected two adjacent relay nodes at the same time.
- Monitoring detected packet loss quickly but did not immediately identify route churn as the likely cause.
- Some mesh nodes retained stale routes longer than expected after rollback.
- The maintenance checklist did not require confirming route preference values before deployment.

## What Went Well

- The outage was detected within minutes.
- The on-call engineer quickly ruled out upstream fiber and gateway failure.
- Rollback was straightforward once the affected nodes were identified.
- Customer support had a current outage message ready and reduced duplicate troubleshooting.

## What Went Poorly

- The change was deployed to redundant relay nodes simultaneously.
- The lab test did not reflect real mesh behavior under multiple competing paths.
- Route churn visibility was spread across several dashboards and required manual correlation.
- Recovery required manual intervention on downstream nodes.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---:|---|
| Add automated validation for Babel route metrics before deployment. | Network Engineering | 2026-06-25 | Open |
| Update rollout procedure to deploy routing changes to one relay per sector at a time. | Network Engineering | 2026-06-21 | Open |
| Add alert for excessive Babel next-hop changes per node. | Observability | 2026-07-02 | Open |
| Build a staging mesh with at least three hops and redundant paths. | Infrastructure | 2026-07-15 | Open |
| Add rollback steps for clearing stale routes to the maintenance checklist. | Operations | 2026-06-22 | Open |
| Review Babel route expiry settings on all relay nodes. | Network Engineering | 2026-06-28 | Open |
| Publish a customer-facing incident summary. | Support | 2026-06-20 | Open |

## Prevention

Future routing changes will be staged through a topology that better represents the production mesh. Deployments will proceed one relay at a time, with route preference checks before and after each step. We will also add monitoring for route churn so that similar failures are identified as routing instability rather than generic packet loss.