# Postmortem: East Harbor Mesh Outage - July 8, 2026

## Summary

On July 8, 2026, East Harbor Mesh experienced a 78-minute service outage affecting a large portion of the north and central mesh. Customers saw full loss of connectivity or severe packet loss after a configuration rollout created a layer-2 loop between the Harbor Hall and Ridgeview rooftop nodes.

The loop caused a broadcast storm across several backhaul links, saturating wireless interfaces and overwhelming two gateway routers. Service was restored after isolating the Ridgeview node and rolling back the bridge configuration.

## Impact

- Incident window: 2026-07-08 19:14-20:32 EDT
- Duration: 78 minutes
- Customers affected: approximately 612
- Full outage: approximately 418 customers
- Degraded service: approximately 194 customers
- Services affected: residential internet, community Wi-Fi hotspots, VoIP adapters
- No customer data loss occurred

## Timeline

All times are EDT.

| Time | Event |
|---|---|
| 19:14 | Automated configuration rollout begins for rooftop relay nodes. |
| 19:17 | Monitoring detects elevated packet loss on Harbor Hall backhaul. |
| 19:20 | First customer reports arrive through SMS support line. |
| 19:23 | NOC dashboard shows DHCP failures and high ARP traffic on north mesh segment. |
| 19:29 | On-call engineer joins incident bridge and pauses remaining rollout. |
| 19:36 | Gateway CPU on `gw-north-01` reaches 100%; BGP session to upstream briefly flaps. |
| 19:44 | Engineers identify abnormal broadcast traffic originating near Ridgeview. |
| 19:52 | Field volunteer confirms Ridgeview node is reachable locally but saturated over wireless. |
| 20:03 | Ridgeview uplink is administratively disabled from Harbor Hall. Packet loss begins dropping. |
| 20:12 | Bridge configuration rollback applied to Harbor Hall and Ridgeview. |
| 20:24 | DHCP success rate and gateway CPU return to normal. |
| 20:32 | Service confirmed restored across affected sectors. |
| 21:05 | Customer status page updated with resolution notice. |

## Root Cause

The outage was caused by a layer-2 loop introduced during a routine configuration rollout.

A new bridge template disabled Spanning Tree Protocol on rooftop relay bridges to reduce convergence delays. This template was applied to Ridgeview, which had both a wireless mesh backhaul and a temporary wired maintenance trunk connected to the same VLAN path through Harbor Hall.

With STP disabled, Ridgeview and Harbor Hall formed an unintended loop. Broadcast and multicast traffic circulated continuously, saturating the wireless backhaul and causing ARP, DHCP, and neighbor discovery traffic to fail across the north and central mesh.

## Contributing Factors

- The rollout validation did not check for nodes with both wired maintenance trunks and active wireless backhaul links.
- The staging environment did not include mixed wired/wireless rooftop topologies.
- Broadcast storm detection existed on gateways but not on intermediate mesh relay nodes.
- The change was scheduled during evening peak usage instead of the normal maintenance window.
- The rollback playbook restored bridge settings but did not automatically clear saturated neighbor tables.

## Detection

The incident was detected through packet loss alerts on the Harbor Hall backhaul and customer reports. Existing monitoring showed gateway CPU saturation and DHCP failures, but it did not immediately identify the layer-2 loop source.

## Resolution

Engineers isolated the Ridgeview node by disabling its Harbor Hall uplink, then rolled back the bridge configuration on both Ridgeview and Harbor Hall. Once the loop stopped, gateway load and DHCP success rates recovered within approximately 20 minutes.

## What Went Well

- The rollout was paused quickly once customer impact was confirmed.
- Field volunteers were able to verify Ridgeview status without requiring roof access.
- The rollback configuration was available and worked as expected.
- Customer support had SMS and status-page channels ready.

## What Went Wrong

- The risky bridge change was not flagged during review.
- Monitoring identified symptoms but not the source node.
- The rollout happened during a high-traffic period.
- Customer-facing updates lagged behind engineering discovery by about 30 minutes.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---:|---|
| Add preflight checks for nodes with mixed wired and wireless paths before bridge changes. | Network Engineering | 2026-07-15 | Open |
| Re-enable STP or equivalent loop protection on all relay bridge templates. | Network Engineering | 2026-07-15 | Open |
| Add broadcast storm alerts at rooftop relay nodes. | Observability | 2026-07-22 | Open |
| Build a staging topology that includes maintenance trunks and redundant mesh paths. | Infrastructure | 2026-07-29 | Open |
| Restrict non-emergency config rollouts during peak evening hours. | Operations | 2026-07-12 | Open |
| Update customer communications process for incidents longer than 15 minutes. | Support | 2026-07-19 | Open |
| Add automated neighbor table cleanup to rollback playbooks. | Network Engineering | 2026-07-22 | Open |

## Follow-Up

A review of all rooftop relay nodes will be completed before the next scheduled network rollout. Future bridge and VLAN changes will require topology-aware validation and approval from both the on-call engineer and the volunteer area lead for the affected mesh segment.