# Postmortem: Transit Planner Routing Outage

**Date:** 2026-06-24  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Service:** Municipal Transit Planner

## Summary

On June 24, 2026, the public transit planner was unable to return trip plans for many riders during the morning commute. The outage affected the web planner, mobile app routing, and third-party consumers of the journey planning API.

The incident was caused by a malformed GTFS schedule bundle that passed validation but triggered a routing graph rebuild failure. The failed rebuild left the planner serving a partially updated graph with missing stop-to-route indexes.

## Customer Impact

From 07:08 to 08:50 local time:

- Approximately 61% of trip planning requests failed or returned incomplete results.
- Riders could search stops and view real-time arrivals, but many origin-to-destination plans returned “No route found.”
- Accessibility routing was unavailable for affected trips.
- Third-party apps using the public journey planning API saw elevated 500 responses.
- Estimated affected users: 38,000 web and mobile users, plus downstream third-party users.

No fare payment, vehicle tracking, or operator dispatch systems were affected.

## Timeline

All times local.

| Time | Event |
| --- | --- |
| 06:45 | Daily GTFS import job downloaded the new schedule bundle from the scheduling system. |
| 06:52 | Static validation completed successfully. |
| 07:03 | Routing graph rebuild started. |
| 07:08 | Error rate began increasing on `/plan-trip` and `/api/v2/journeys`. |
| 07:12 | Synthetic monitoring alerted on elevated planner failures. |
| 07:17 | On-call engineer acknowledged the alert and confirmed widespread routing failures. |
| 07:25 | Team identified that failures started after the morning graph rebuild. |
| 07:34 | Real-time arrivals and stop search were confirmed healthy; incident narrowed to journey planning graph data. |
| 07:42 | Attempted graph rebuild from the latest GTFS bundle failed with missing transfer references. |
| 07:55 | Team rolled back to the previous day’s known-good routing graph. |
| 08:11 | Planner API error rate began recovering, but cache nodes continued serving partial results. |
| 08:27 | Edge and application caches were purged. |
| 08:50 | Error rate returned to baseline and trip planning was fully restored. |
| 09:15 | Incident declared resolved. |
| 10:30 | Root cause review began with Scheduling Systems and Transit Data teams. |

## Root Cause

A schedule export contained transfer records referencing two stops that had been removed from `stops.txt` during a late service change. Our GTFS validator checked file format, required fields, and referential integrity for trips and stop times, but did not validate transfer references against removed stops.

During graph generation, the router skipped the invalid transfer records but still committed the partially rebuilt stop-to-route index. This created an internally inconsistent graph:

- Route patterns loaded successfully.
- Some transfer edges were missing.
- Several stop indexes were incomplete.
- Trip planning requests involving affected stops failed or returned no viable route.

The deployment pipeline treated the graph build as successful because the process exited with status `0` after logging recoverable warnings.

## Contributing Factors

- GTFS validation did not cover all reference relationships.
- Graph rebuilds were not atomic from the planner service’s perspective.
- Build warnings were not promoted to failures for invalid transfer references.
- The rollback process restored graph files but did not automatically purge application caches.
- Synthetic monitoring detected failures, but no alert existed for “no route found” spikes.

## Resolution

The team restored the previous known-good routing graph, purged stale planner caches, and disabled automatic promotion of new schedule bundles until additional validation was added.

A corrected GTFS bundle was generated later that morning and manually promoted after validation.

## Action Items

| Action | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Add referential integrity checks for `transfers.txt` stop IDs. | Transit Data | High | 2026-07-01 |
| Make routing graph promotion atomic with explicit success markers. | Platform | High | 2026-07-08 |
| Fail graph builds on invalid GTFS references instead of logging warnings. | Routing | High | 2026-07-08 |
| Add automated cache purge to graph rollback workflow. | Platform | Medium | 2026-07-15 |
| Alert on abnormal increases in “No route found” responses. | SRE | Medium | 2026-07-15 |
| Add pre-publish validation in the scheduling export pipeline. | Scheduling Systems | High | 2026-07-22 |
| Run a quarterly rollback drill for planner data updates. | SRE | Low | 2026-08-15 |

## What Went Well

- Synthetic monitoring detected the outage within minutes.
- The previous day’s routing graph was available and restorable.
- Teams quickly narrowed the issue to schedule data rather than vehicle telemetry or API infrastructure.

## What Could Be Improved

- Invalid schedule data should have been rejected before graph generation.
- Partial graph builds should never be visible to production services.
- Rollback should restore both data and cache state as a single operation.
- Customer communications should have started earlier during the morning commute window.