# Postmortem: Trip Planning Outage on July 24, 2026

**Incident date:** July 24, 2026  
**Duration:** 08:12–10:06 EDT  
**Severity:** SEV-1  
**Services affected:** Web trip planner, mobile applications, public journey-planning API  
**Status:** Resolved

## Summary

On July 24, the municipal transit trip-planning service was unavailable or returned incomplete itineraries for approximately 1 hour and 54 minutes during the morning commute.

A scheduled General Transit Feed Specification (GTFS) import introduced duplicate stop identifiers into the production routing dataset. The routing engine rejected the new dataset and attempted to retain the previous version. However, a deployment change made earlier that week had moved dataset validation from before activation to after activation. As a result, the previous dataset was removed before the replacement was confirmed usable.

Automatic recovery repeatedly restarted routing instances with the same invalid dataset. The incident ended after engineers restored the prior dataset from object storage, disabled the import job, and restarted the routing fleet.

## Customer Impact

Between 08:12 and 10:06 EDT:

- Approximately 184,000 trip-planning requests failed.
- 71% of API requests returned HTTP 503 responses.
- 18% returned incomplete itineraries that omitted bus connections.
- The web planner displayed “Trip planning is temporarily unavailable.”
- The iOS and Android applications continued to show saved trips and real-time arrival boards, but customers could not create new multimodal journeys.
- Third-party accessibility and commuter applications using the public API were also affected.
- Vehicle tracking, fare payment, station displays, and transit operations systems were not affected.

The outage occurred during the morning commute and had a disproportionate impact on riders making transfers or planning unfamiliar and accessible journeys.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 07:55 | The scheduled GTFS import begins after publication of the weekday service update. |
| 08:08 | The importer completes transformation and publishes dataset version `2026-07-24.1` to production storage. |
| 08:10 | Routing instances begin loading the new dataset. The active dataset pointer is updated before validation finishes. |
| 08:12 | The first routing instance exits after detecting duplicate stop identifiers. Load balancer health checks begin removing instances from service. |
| 08:14 | Error rate exceeds 20%. A warning alert is sent to the trip-planning team’s chat channel. |
| 08:17 | Kubernetes restarts failed instances. Each restart loads the same invalid dataset and fails again. |
| 08:21 | Available routing capacity falls below demand. Customers begin receiving widespread HTTP 503 responses. |
| 08:25 | The on-call engineer acknowledges the alert and begins investigating a suspected capacity issue. |
| 08:31 | Autoscaling increases the routing fleet from 24 to 48 instances, raising database and object-storage traffic without restoring capacity. |
| 08:38 | API error rate reaches 71%. The incident is escalated to SEV-1, and an incident commander is assigned. |
| 08:43 | Engineers identify dataset-loading failures in routing-engine logs. |
| 08:49 | The GTFS import job is disabled to prevent further dataset publication. |
| 08:54 | Engineers attempt to roll back the application deployment. The rollback completes but does not restore service because the active dataset pointer still references the invalid version. |
| 09:06 | The team confirms that the previously active local dataset was deleted during activation. |
| 09:17 | The prior dataset, version `2026-07-23.3`, is located in object-storage history and prepared for restoration. |
| 09:28 | Restoration begins. A checksum mismatch caused by an incomplete download delays activation on six instances. |
| 09:41 | The dataset is downloaded again and validated successfully on a canary instance. |
| 09:47 | The active dataset pointer is changed to `2026-07-23.3`, and the routing fleet is restarted in batches. |
| 09:55 | More than 80% of routing instances are healthy. Error rates decline rapidly. |
| 10:02 | Web, mobile, and public API trip-planning checks pass across all supported travel modes. |
| 10:06 | Error rates return to baseline. The incident is declared resolved. |
| 11:20 | A corrected GTFS feed is received from the scheduling team. Publication remains disabled pending additional validation. |

## Root Cause

The immediate cause was an invalid GTFS dataset containing duplicate stop identifiers for 14 temporary bus stops introduced as part of a construction detour. The feed passed the existing schema checks because those checks verified field presence and format but did not enforce identifier uniqueness across merged source files.

The outage occurred because the production activation workflow updated the active dataset pointer and deleted the previous local dataset before the routing engine completed semantic validation.

This behavior was introduced in a deployment three days earlier. The change was intended to reduce disk usage and shorten dataset activation time. It altered the workflow from:

1. Download the candidate dataset.
2. Validate and load it.
3. Switch traffic to the new dataset.
4. Remove the previous dataset.

to:

1. Download the candidate dataset.
2. Mark it active.
3. Remove the previous dataset.
4. Validate and load it.

When validation failed, routing instances restarted and repeatedly selected the invalid active version. Because the valid prior version had already been removed locally, the instances could not fall back automatically.

## Contributing Factors

- GTFS validation checked structure but not uniqueness across merged feeds.
- Dataset activation and cleanup were implemented as one non-transactional operation.
- Health checks reported an instance as ready before the routing graph was fully loaded.
- Restart automation retried indefinitely without quarantining the failed dataset version.
- The initial alert was classified as a warning and did not page the on-call engineer until customer error rates crossed a separate threshold.
- The operational runbook described application rollback but not dataset rollback.
- Autoscaling treated failed health checks as insufficient capacity, increasing restart traffic and log volume.
- No end-to-end canary trip plans were required before fleet-wide dataset activation.

## Resolution and Recovery

Engineers disabled the import job, restored the previous GTFS dataset from object-storage history, validated it on a canary instance, and restarted the routing fleet in controlled batches.

Service recovered as healthy instances completed graph loading and rejoined the load balancer. The team kept the new feed disabled until the scheduling group supplied a corrected version and additional uniqueness checks were added.

## What Went Well

- Object-storage version history preserved the last valid dataset.
- Vehicle tracking and fare systems were isolated from the trip-planning platform.
- Cross-team escalation brought scheduling and platform engineers into the incident quickly.
- Synthetic checks covering bus, rail, and accessible journeys confirmed recovery before the incident was closed.

## What Went Poorly

- The deployment change modified failure behavior without a corresponding rollback test.
- Monitoring detected infrastructure symptoms before identifying dataset activation as the cause.
- The service could not automatically retain or restore the last known-good dataset.
- The first rollback targeted application code rather than the active data version, extending recovery time.
- Public status updates began 27 minutes after widespread customer impact.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Make dataset activation atomic and retain the last two validated versions locally. | Routing Platform | P0 | August 7, 2026 |
| Add uniqueness and referential-integrity checks across merged GTFS feeds. | Transit Data | P0 | August 4, 2026 |
| Require successful graph construction before updating the active dataset pointer. | Routing Platform | P0 | August 7, 2026 |
| Add automatic fallback to the last known-good dataset after a load failure. | Routing Platform | P0 | August 14, 2026 |
| Quarantine rejected dataset versions to prevent restart loops. | Infrastructure | P1 | August 14, 2026 |
| Change readiness checks to remain false until the routing graph is loaded and test queries pass. | Infrastructure | P1 | August 10, 2026 |
| Add canary validation for representative bus, rail, transfer, and accessible journeys. | Quality Engineering | P1 | August 21, 2026 |
| Page immediately when more than 10% of trip-planning requests fail for five minutes. | Observability | P1 | August 5, 2026 |
| Prevent autoscaling from responding to dataset-load failures. | Infrastructure | P1 | August 21, 2026 |
| Document and automate dataset rollback in the incident runbook. | Service Operations | P1 | August 7, 2026 |
| Add a deployment test covering invalid dataset activation and recovery. | Routing Platform | P1 | August 14, 2026 |
| Establish a 15-minute target for publishing customer-facing status updates during SEV-1 incidents. | Communications | P2 | August 28, 2026 |

## Lessons Learned

Routing datasets are production artifacts and require the same validation, staged rollout, and rollback protections as application code. A dataset should never become active until it has been fully validated and loaded, and cleanup must not remove the last known-good version until the replacement is proven healthy.

The team will treat future feed publication changes as high-risk production changes, with canary activation, automated rollback, and explicit monitoring of data-version health.