# Postmortem: Transit Planner Trip Search Outage

**Date:** 2026-07-08  
**Duration:** 1 hour 42 minutes  
**Services affected:** Web trip planner, mobile trip planning API, station departure boards using planner-backed routes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 8, 2026, the municipal transit planner was unable to return trip plans for most users during the morning commute. Users could load the application, view maps, and check some static route pages, but trip searches either timed out or returned “No routes found.”

The outage was caused by a malformed schedule import that triggered excessive graph rebuild work across all planner instances. The rebuilds saturated CPU and memory, causing API requests to queue and eventually fail.

## Customer Impact

From **07:14 to 08:56 EDT**:

- Approximately **82% of trip planning requests failed**.
- Average API latency increased from **280 ms to over 18 seconds**.
- Mobile and web users could not reliably plan trips.
- Real-time arrival data remained available, but connections and multi-leg itineraries were unavailable.
- Customer support received **1,184 outage-related contacts**.
- The outage affected an estimated **64,000 commuters** during peak travel time.

## Timeline

All times are EDT.

| Time | Event |
|---|---|
| 06:45 | Daily GTFS schedule import completed successfully according to the import job. |
| 07:09 | Planner instances began rebuilding routing graphs after detecting new schedule data. |
| 07:14 | API latency alert fired for `/trip-plan`. |
| 07:18 | On-call engineer acknowledged the alert. |
| 07:23 | Error rate exceeded 50%; incident declared SEV-1. |
| 07:31 | Initial investigation focused on database latency; no database issue found. |
| 07:44 | CPU saturation observed across all planner instances. |
| 07:52 | Rolling restart attempted, but new instances immediately entered graph rebuild loops. |
| 08:07 | Schedule import identified as the common trigger. |
| 08:16 | Team found malformed stop sequence data in one route block from the latest feed. |
| 08:24 | Schedule feed was rolled back to the previous known-good version. |
| 08:37 | Planner instances completed graph rebuild using the previous feed. |
| 08:48 | Error rate dropped below 5%. |
| 08:56 | Service confirmed stable; incident resolved. |
| 09:20 | Public status page updated with full resolution notice. |

## Root Cause

The root cause was a malformed GTFS schedule record for a temporary bus detour route. The record contained a duplicated stop sequence with conflicting arrival times.

The schedule validator detected the duplicate sequence but classified it as a warning rather than a blocking error. During graph construction, the planner repeatedly attempted to reconcile the conflicting stop order. This caused graph rebuilds to consume excessive CPU and memory. Because all planner instances consumed the same imported feed, the failure affected the entire fleet.

Contributing factors:

- Feed validation allowed malformed stop sequences into production.
- Planner instances rebuilt routing graphs simultaneously instead of staggering rebuilds.
- The graph builder did not enforce a maximum retry or fail-fast condition for invalid stop ordering.
- The alert identified API latency but did not immediately point responders toward schedule import health.

## Resolution

The operations team rolled back the active schedule feed to the previous known-good version. Planner instances were restarted after the rollback and successfully rebuilt routing graphs. Trip planning service returned to normal once the rebuild completed.

## What Went Well

- On-call response began within 4 minutes of the first alert.
- The previous schedule feed was available and could be restored quickly.
- Real-time arrival and static route information remained online.
- Customer support and communications teams were engaged during the incident.

## What Went Wrong

- The import pipeline marked the malformed feed as successful.
- All planner instances failed in the same way at the same time.
- Restarting instances before identifying the feed issue increased recovery time.
- The public status update lagged behind the internal incident declaration.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Reclassify duplicated stop sequences with conflicting times as blocking validation errors. | Data Platform | P0 | 2026-07-15 |
| Add graph builder fail-fast behavior for invalid stop ordering. | Routing Team | P0 | 2026-07-18 |
| Keep the previous routing graph loaded until a new graph is fully built and verified. | Routing Team | P0 | 2026-07-25 |
| Stagger graph rebuilds across planner instances. | Infrastructure | P1 | 2026-07-26 |
| Add alerts for schedule import warnings exceeding threshold. | SRE | P1 | 2026-07-22 |
| Update incident runbook to check latest schedule import during planner-wide failures. | SRE | P1 | 2026-07-19 |
| Automate public status page update when SEV-1 is declared. | Communications Platform | P2 | 2026-08-02 |

## Prevention

The highest-impact prevention work is to reject invalid schedule feeds before they reach production and to preserve the last known-good routing graph during rebuild failures. With those changes, a malformed feed should result in delayed schedule publication rather than a user-facing trip planning outage.