# Postmortem: North Sector Mesh Outage

**Date:** 2026-07-10  
**Duration:** 2h 47m  
**Status:** Resolved  
**Severity:** SEV-2  
**Affected area:** North sector community mesh, including Hilltop, East Market, and Cedar blocks

## Summary

On 2026-07-10, a routing loop and backhaul saturation caused intermittent or complete loss of internet access for customers connected through the North sector mesh. The outage began after a planned firmware upgrade on rooftop node `north-gw-03`. The upgrade reset a wireless bridge interface to its default metric, causing mesh traffic to prefer a low-capacity fallback path.

The fallback link saturated, routing reconverged repeatedly, and several downstream nodes lost stable default routes. Service was restored after manually correcting the bridge metric, clearing stale routes, and rate-limiting management traffic during convergence.

## Customer Impact

- Approximately 410 customer households affected
- 165 customers experienced complete loss of connectivity
- Remaining affected customers saw high packet loss, latency above 2 seconds, or repeated disconnects
- VoIP service was unreliable for affected customers
- Local mesh services remained reachable for most nodes, but internet access was degraded or unavailable

## Timeline

All times are in Eastern Time.

| Time | Event |
|---|---|
| 09:00 | Scheduled maintenance window begins for firmware updates on North sector rooftop nodes. |
| 09:18 | `north-gw-03` firmware upgrade completes and node rejoins the mesh. |
| 09:23 | Monitoring detects increased latency on `north-backhaul-a`. |
| 09:31 | First customer reports connectivity loss in Hilltop block. |
| 09:38 | NOC observes route churn between `north-gw-03`, `north-relay-07`, and `cedar-roof-02`. |
| 09:52 | Packet loss exceeds 40% across North sector customer gateways. Incident declared SEV-2. |
| 10:11 | Engineers identify that traffic is preferring the 80 Mbps fallback link instead of the 1 Gbps primary bridge. |
| 10:27 | Firmware configuration diff shows bridge metric reset on `north-gw-03`. |
| 10:39 | Correct metric applied manually and routing daemon restarted on `north-gw-03`. |
| 10:48 | Route churn decreases, but several downstream nodes keep stale routes. |
| 11:05 | Stale routes cleared on affected relay nodes. |
| 11:22 | Packet loss returns below 2%; customer sessions begin recovering. |
| 11:47 | Incident resolved after 25 minutes of stable monitoring. |

## Root Cause

The root cause was a firmware upgrade on `north-gw-03` that reset the metric on the primary wireless bridge interface from `10` to the default value of `600`. This made the routing daemon prefer a lower-capacity fallback path through `north-relay-07`.

The fallback path was intended only for emergency connectivity and could not carry normal sector traffic. Once saturated, routing updates were delayed or dropped, causing repeated route recalculation and unstable default routes for downstream nodes.

## Contributing Factors

- The firmware upgrade process did not preserve custom interface metrics.
- Pre-maintenance checks validated node reachability but did not validate expected route preference.
- The fallback link had no capacity guardrail or traffic shaping.
- Alerting detected symptoms quickly but did not identify route preference changes directly.
- Runbooks did not include a post-upgrade mesh metric verification step.

## Resolution

Engineers restored service by:

1. Reapplying the correct bridge metric on `north-gw-03`.
2. Restarting the routing daemon to force route recalculation.
3. Clearing stale routes on downstream relay nodes.
4. Temporarily rate-limiting management and telemetry traffic during recovery.
5. Verifying stable default routes and packet loss across affected mesh segments.

## What Went Well

- Customer reports and monitoring aligned quickly, helping confirm the affected area.
- Engineers isolated the problem to a specific gateway within one hour.
- The fallback path prevented a wider hard outage, even though it became saturated.
- Manual recovery steps were effective once the incorrect metric was identified.

## What Went Poorly

- The planned maintenance did not include validation of routing metrics after firmware upgrade.
- The fallback path failed in a noisy way, causing route churn instead of graceful degradation.
- Customer support did not receive a clear incident summary until 54 minutes after declaration.
- Several relay nodes required manual intervention because stale routes persisted longer than expected.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---|---|
| Add automated post-upgrade validation for bridge metrics and preferred routes. | Network Engineering | High | 2026-07-24 |
| Update firmware upgrade runbook to include route preference checks before closing maintenance. | Operations | High | 2026-07-19 |
| Add alert for unexpected traffic volume on fallback backhaul links. | Observability | Medium | 2026-07-31 |
| Apply traffic shaping to fallback links to preserve routing and management traffic during saturation. | Network Engineering | High | 2026-08-02 |
| Build config backup and restore check into rooftop node upgrade workflow. | Platform | Medium | 2026-08-09 |
| Create customer support incident template for sector-level outages. | Support | Medium | 2026-07-22 |
| Test routing convergence behavior in the lab using current production firmware. | Network Engineering | Medium | 2026-08-16 |

## Prevention

Future firmware upgrades on mesh gateways will require automated validation of route metrics, backhaul preference, and fallback path utilization before the maintenance window can be closed. Fallback links will also be configured to preserve control-plane traffic under saturation so that degraded service does not cascade into unstable routing.