# Postmortem: Gateway Routing Failure and Partial Network Outage

**Incident date:** 2026-07-18  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Affected service:** Community mesh internet access  
**Primary region:** Eastside mesh  
**Authors:** Network Operations Team

## Summary

On July 18, the Eastside community mesh experienced a partial outage after a routine configuration deployment caused both regional gateway routers to stop advertising a valid default route.

Mesh connectivity within individual neighborhoods remained available, but most members could not reach the public internet. Some traffic continued through a lower-capacity gateway in the adjacent Central region, resulting in high latency, packet loss, and intermittent connectivity.

The incident began at 19:06 and ended at 21:53 after operators restored the previous routing configuration and reset affected BGP sessions. Approximately 1,840 member connections were disrupted.

## Customer Impact

From 19:06 to 21:53:

- Approximately 1,310 members lost internet access entirely.
- Approximately 530 members experienced intermittent connectivity, elevated latency, or packet loss.
- Local mesh services, including the community message board and neighborhood file mirrors, remained reachable when both endpoints were on the same mesh segment.
- VoIP calls frequently failed or disconnected.
- Members using the adjacent Central gateway saw median latency increase from 24 ms to 310 ms, with packet loss peaking at 38%.
- The public status page was initially unavailable because it was hosted behind the affected gateways.

No customer data was lost or exposed.

## Detection

Automated monitoring detected a drop in successful external probes at 19:08. The alert reached the on-call operator at 19:10.

The initial alert identified general internet reachability failure but did not indicate that the default route had been withdrawn. Operators first investigated upstream-provider connectivity and gateway hardware before examining the routing-policy deployment.

Reports from members began arriving through the community chat channel at 19:12.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 18:42 | An operator approved a routing-policy change intended to reject invalid upstream prefixes. |
| 18:58 | The configuration deployment began on Eastside Gateway A. |
| 19:02 | Gateway A reloaded its BGP policy. Its upstream default route was rejected. |
| 19:04 | Automated validation checked that the BGP session was established but did not verify that a usable default route existed. The deployment continued. |
| 19:06 | Gateway B received the same configuration and withdrew its default route. Member traffic began failing or shifting to the Central gateway. |
| 19:08 | External reachability probes crossed the alert threshold. |
| 19:10 | The on-call operator acknowledged the alert. |
| 19:12 | Member reports appeared in the community chat channel. |
| 19:18 | Operators confirmed that both Eastside gateways were reachable and that their upstream interfaces were active. |
| 19:27 | The team contacted the upstream provider to investigate a possible transit failure. |
| 19:39 | The upstream provider confirmed that both circuits and BGP sessions were operational. |
| 19:47 | Operators observed that neither gateway had installed a default route. |
| 19:55 | The team identified the routing-policy deployment as the likely cause. |
| 20:04 | A rollback was attempted on Gateway A. The configuration reverted, but the BGP session did not automatically re-evaluate previously rejected routes. |
| 20:19 | Operators manually reset Gateway A’s upstream BGP session. A valid default route was restored. |
| 20:23 | Traffic began returning through Gateway A, which became overloaded because Gateway B remained unavailable. |
| 20:31 | Rate limits were applied to nonessential community mirror traffic to stabilize Gateway A. |
| 20:46 | Gateway B was rolled back. |
| 20:52 | Gateway B failed to restore routes because the rollback package contained an outdated peer address. |
| 21:11 | Operators corrected the peer address and re-established the BGP session. |
| 21:18 | Both gateways were advertising valid default routes, and packet loss began returning to normal. |
| 21:37 | Monitoring showed stable connectivity across all Eastside mesh segments. |
| 21:53 | The incident was declared resolved after a 15-minute validation period. |

## Root Cause

The outage was caused by an incorrect routing-policy rule deployed to both Eastside gateways.

The change was intended to reject overly broad route announcements from transit peers while explicitly permitting the IPv4 and IPv6 default routes. However, the generated configuration placed the general rejection rule before the default-route exceptions. Because the router evaluates these policy statements in order, the default routes matched the rejection rule and were discarded before reaching the exceptions.

Both gateways remained connected to the upstream provider, and their BGP sessions stayed in the established state. This made the failure appear healthy to the deployment system even though neither router had a usable route to the public internet.

The deployment proceeded automatically from Gateway A to Gateway B because its validation step checked only interface status, BGP session state, and configuration syntax. It did not test the routing table or end-to-end internet reachability.

## Contributing Factors

- Both regional gateways received the change within four minutes, leaving insufficient time to observe customer-facing effects on Gateway A.
- The policy generator had unit tests for individual rules but no test for final rule ordering.
- Predeployment validation used a synthetic configuration that did not include a default route.
- The deployment health check treated an established BGP session as proof of working transit.
- The adjacent Central gateway accepted overflow traffic but lacked enough capacity for the full Eastside region.
- The status page was hosted inside the affected network and was unavailable during the first 41 minutes of the incident.
- The rollback procedure restored configuration files but did not reset BGP sessions or verify route recovery.
- Gateway B’s rollback package contained stale peer information because rollback artifacts were not regenerated after a previous circuit migration.
- Volunteer operators lacked a single runbook for distinguishing upstream circuit failures from routing-policy failures.

## Resolution and Recovery

Operators restored service by reverting the routing policy on both gateways, correcting the stale peer address on Gateway B, and manually resetting the upstream BGP sessions.

Once valid default routes were installed, traffic shifted away from the overloaded Central gateway. Temporary rate limits on community mirror traffic reduced congestion while Gateway B was being recovered. The limits were removed after both Eastside gateways had remained stable for 30 minutes.

## What Went Well

- External probes detected the outage within two minutes.
- Community members quickly reported symptoms and identified neighborhoods with partial connectivity.
- Local mesh services continued operating during the internet outage.
- The upstream provider responded promptly and helped rule out circuit failure.
- Configuration versioning made it possible to identify the triggering change and restore the previous policy.

## What Went Poorly

- The deployment system promoted a broken configuration to the redundant gateway.
- Monitoring did not distinguish between BGP session health and usable route availability.
- The first rollback did not fully restore service without manual intervention.
- The status page shared infrastructure with the affected service.
- Capacity planning assumed that the Central gateway would absorb only a small portion of Eastside traffic.
- Member communications were delayed because responders focused on diagnosis before assigning a communications lead.

## Corrective and Preventive Actions

| Action | Owner | Priority | Due date | Status |
|---|---|---:|---|---|
| Add checks for installed IPv4 and IPv6 default routes to deployment validation. | Network Automation | P0 | 2026-07-22 | Completed |
| Require successful external HTTP, DNS, and ICMP probes before deploying to the second gateway. | Network Automation | P0 | 2026-07-25 | Completed |
| Introduce a minimum 20-minute observation period between redundant gateway deployments. | Network Operations | P0 | 2026-07-22 | Completed |
| Add integration tests that validate complete generated routing-policy order. | Platform Engineering | P1 | 2026-08-08 | In progress |
| Regenerate and test rollback artifacts after every peer or circuit change. | Network Operations | P1 | 2026-08-05 | In progress |
| Update rollback automation to reset affected BGP sessions and verify route installation. | Network Automation | P1 | 2026-08-12 | Planned |
| Add alerts for missing default routes, abnormal route-count changes, and unexpected gateway failover. | Observability | P1 | 2026-08-09 | In progress |
| Move the public status page to infrastructure outside the community mesh. | Infrastructure | P1 | 2026-08-15 | Planned |
| Increase Central gateway capacity to support at least 70% of Eastside peak traffic. | Capacity Working Group | P2 | 2026-09-30 | Planned |
| Publish a routing-failure diagnostic runbook for volunteer operators. | Network Operations | P1 | 2026-08-03 | In progress |
| Assign an incident communications lead at the start of every SEV-1 incident. | Operations Committee | P1 | 2026-08-01 | Completed |
| Run a quarterly gateway-failover exercise involving operators and community coordinators. | Reliability Working Group | P2 | 2026-09-15 | Planned |

## Lessons Learned

Redundant hardware did not protect the network because both gateways received the same faulty configuration in rapid succession. Gateway redundancy must include deployment isolation, independent validation, and sufficient time to detect failures before changes are promoted.

We also relied too heavily on control-plane indicators. An established routing session does not guarantee that customers can reach the internet. Future health checks will validate installed routes and customer-visible connectivity as separate requirements.

Finally, recovery procedures must be tested as complete workflows. Restoring a configuration file was insufficient because route re-evaluation and peer-state validation were also required. Quarterly failover exercises will include rollback testing under realistic traffic conditions.