# Postmortem: East Ridge Mesh Outage

**Incident date:** 2026-07-08  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Authors:** Network Operations Team

## Summary

On July 8, East Ridge Community Mesh experienced a partial outage affecting subscribers connected through the Hilltop relay cluster. The outage began after an automatic firmware upgrade caused several relay nodes to reboot simultaneously and fail to rejoin the mesh due to an incompatible routing daemon configuration.

Traffic for affected neighborhoods could not reroute because two redundant backhaul paths were already degraded from unrelated radio alignment issues. Service was restored after operators rolled back the firmware on the affected relay nodes and manually restarted the routing daemon.

## Customer Impact

Approximately 430 subscribers experienced degraded or unavailable internet service.

Impact included:

- Complete loss of connectivity for about 290 subscribers in East Ridge, Oak Hollow, and the north side of Mill Street
- High packet loss and intermittent connectivity for about 140 additional subscribers
- Increased latency across adjacent mesh segments due to failed rerouting attempts
- Delayed support responses because the outage also affected one volunteer-run support station

No customer data was lost or exposed.

## Timeline

All times are local.

| Time | Event |
| --- | --- |
| 18:04 | Automated firmware rollout begins for Hilltop relay cluster. |
| 18:11 | First relay node reboots and fails to rejoin the mesh. |
| 18:16 | Monitoring detects packet loss above 40% on East Ridge sector links. |
| 18:21 | Additional relay nodes reboot, causing loss of primary path through Hilltop. |
| 18:28 | Pager alerts network operations volunteer on call. |
| 18:35 | Operators confirm that Hilltop relay nodes are reachable over management Wi-Fi but are not advertising routes. |
| 18:47 | Attempted service restart fails because the new firmware expects a different routing daemon config path. |
| 19:03 | Operators identify recent firmware upgrade as likely trigger. |
| 19:18 | Firmware rollout is paused globally. |
| 19:31 | Manual rollback begins on the first Hilltop relay node. |
| 19:48 | First node rejoins the mesh and begins advertising routes. |
| 20:04 | Remaining affected nodes are rolled back. |
| 20:16 | Customer traffic starts returning to normal levels. |
| 20:21 | Monitoring shows packet loss below 2% across affected sectors. |
| 20:28 | Incident declared resolved. |

## Root Cause

The immediate root cause was a firmware upgrade that changed the expected configuration location for the routing daemon. Our deployment process updated the firmware but did not migrate the existing routing configuration or validate that the daemon successfully advertised routes after reboot.

The broader contributing factors were:

- The rollout allowed multiple relay nodes in the same cluster to reboot within the same maintenance window.
- Pre-upgrade validation checked node reachability but not route advertisement health.
- Two backup backhaul paths had known signal degradation and could not carry the failed cluster's traffic.
- Alerting detected packet loss but did not clearly identify the missing mesh routes as the failure mode.

## What Went Well

- Monitoring detected the customer-facing impact within minutes.
- Management access remained available, allowing remote rollback.
- The firmware rollout was paused before other neighborhoods were affected.
- Volunteers in the field confirmed local radio and power conditions quickly.

## What Went Poorly

- The upgrade was not canaried on a single relay node before cluster-wide rollout.
- The deployment tool treated reboot completion as success.
- Known backhaul degradation reduced redundancy during the incident.
- Customer status updates were delayed while operators focused on diagnosis.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add post-upgrade validation for routing daemon health and route advertisement. | Network Automation | 2026-07-18 | Open |
| Change firmware rollout policy to upgrade one relay per mesh cluster at a time. | Network Operations | 2026-07-19 | Open |
| Add automatic rollback when upgraded nodes fail route health checks. | Network Automation | 2026-07-25 | Open |
| Repair and realign degraded backup backhaul links. | Field Team | 2026-07-22 | Open |
| Add alerts for missing route advertisements by relay cluster. | Observability | 2026-07-24 | Open |
| Create an incident communication checklist for volunteer operators. | Support Team | 2026-07-20 | Open |
| Document firmware compatibility checks before future rollouts. | Network Operations | 2026-07-21 | Open |

## Prevention

Future firmware upgrades will be staged by cluster and gated on service-level checks, not just device availability. A node will only be considered successfully upgraded if it rejoins the mesh, advertises expected routes, passes packet-loss checks, and carries test traffic through at least one upstream path.

We will also treat degraded backup links as active reliability risks rather than maintenance backlog. Redundant paths must be healthy before planned work on primary relay infrastructure.