# Postmortem: Trip Planning Service Outage — July 18, 2026

**Incident date:** July 18, 2026  
**Duration:** 08:12–10:47 EDT  
**Severity:** SEV-1  
**Services affected:** Web trip planner, mobile applications, public journey-planning API  
**Status:** Resolved

## Summary

On July 18, the municipal transit trip-planning service experienced a 2-hour, 35-minute outage during the morning travel period. Riders could not reliably search for routes, departure times, or accessible journeys. Most requests returned an error or timed out; live vehicle maps and service-alert pages remained available.

The outage began after a routine schedule-data import activated a malformed transfer record. A validation gap allowed the record into production, where it caused routing workers to repeatedly exhaust memory while rebuilding the journey graph. Automated restarts increased pressure on the shared cache and database, preventing healthy workers from recovering.

Service was restored by rolling back the schedule dataset, clearing the corrupted graph cache, and temporarily limiting graph rebuild concurrency.

## Customer Impact

Between 08:12 and 10:47 EDT:

- Approximately 184,000 trip-planning requests failed or timed out.
- The request success rate fell from 99.8% to a low of 11%.
- Median response time increased from 420 milliseconds to 18 seconds.
- Web and mobile users saw “Unable to plan this trip” or indefinite loading states.
- Third-party applications using the public journey-planning API received HTTP 502, 503, or timeout responses.
- Call-centre volume increased by 63% compared with a typical Saturday morning.
- Paratransit booking and dispatch systems were not affected, but customers could not use the public planner to compare accessible fixed-route options.

No customer accounts, payment systems, vehicle-control systems, or personal data were affected.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 07:55 | The automated schedule pipeline imports a revised dataset for weekend construction detours. |
| 08:03 | Schema and referential-integrity checks pass. The dataset is approved automatically for activation. |
| 08:08 | Production routing workers begin rebuilding their journey graphs from the new dataset. |
| 08:12 | The first worker exceeds its memory limit and restarts. Trip-planning error rates begin rising. |
| 08:15 | Monitoring detects elevated HTTP 5xx responses and pages the on-call engineer. |
| 08:20 | Error rate reaches 47%. Kubernetes restarts multiple routing workers after memory-limit violations. |
| 08:27 | The incident is declared SEV-1. The incident commander, operations lead, and transit-data team are engaged. |
| 08:34 | Engineers initially suspect an application deployment from the previous evening and begin comparing versions. |
| 08:46 | The team confirms there was no correlation with application version or traffic volume. Memory growth is isolated to graph rebuilds. |
| 08:55 | Automatic restarts create simultaneous graph rebuilds, saturating the shared cache and increasing database load. Success rate falls to 11%. |
| 09:04 | A public service notice is posted on the transit website and social channels. |
| 09:12 | Engineers disable automatic schedule activation and stop additional routing-worker restarts. |
| 09:26 | Heap analysis identifies an unexpectedly large transfer-edge collection associated with one station complex. |
| 09:38 | The transit-data team finds a malformed transfer record with a circular parent-station reference. |
| 09:51 | The decision is made to roll back to the previous schedule dataset. |
| 10:05 | Rollback begins. The corrupted journey-graph cache is cleared. |
| 10:18 | Routing workers restart with graph-build concurrency limited to one worker per cluster. |
| 10:31 | Success rate recovers to 92%; latency remains elevated while caches warm. |
| 10:42 | Success rate exceeds 99%, and response times return to normal. |
| 10:47 | The incident is resolved. Monitoring continues under heightened observation. |
| 12:30 | The corrected schedule dataset passes manual validation and is deployed successfully. |

## Root Cause

The direct cause was a malformed transfer record in the schedule dataset. The record assigned a station complex as its own parent through an indirect circular reference. During graph construction, the routing engine expanded transfers recursively without detecting that it had already visited the same station hierarchy.

This produced millions of duplicate transfer edges and caused routing workers to exceed their 8 GB memory limits. Kubernetes restarted the failed workers as designed, but each replacement immediately attempted the same graph rebuild.

The restart behavior amplified the incident. Multiple workers rebuilt graphs concurrently, placing heavy load on the shared cache and schedule database. Workers that had previously loaded a healthy graph began timing out while accessing those dependencies, expanding the outage from graph-building workers to the entire trip-planning fleet.

## Contributing Factors

- The schedule pipeline validated record structure and foreign-key existence but did not check station-parent relationships for cycles.
- The routing engine had no maximum traversal depth or visited-node protection for transfer hierarchy expansion.
- Schedule datasets were activated across all production workers without a canary phase.
- Graph rebuilds were allowed to run concurrently across the full worker fleet.
- Restart policies did not distinguish between transient failures and deterministic failures caused by input data.
- Alerts identified elevated errors and memory usage but did not immediately correlate them with the schedule activation.
- The rollback procedure required separate manual steps for dataset selection, cache invalidation, and worker restart.
- The public status page depended on a manual update, delaying customer communication by approximately 49 minutes.

## Resolution and Recovery

The incident team restored service by:

1. Stopping automatic activation of the new schedule dataset.
2. Rolling production back to the last known-good dataset.
3. Clearing cached journey graphs built from the malformed data.
4. Limiting graph rebuild concurrency to reduce cache and database pressure.
5. Restarting routing workers in small batches and verifying request success before continuing.
6. Correcting the circular station reference and manually validating the revised dataset before deployment.

## What Went Well

- Error-rate monitoring detected the incident within three minutes.
- On-call engineering, transit-data operations, and customer communications joined the incident promptly.
- Historical schedule datasets were retained and could be restored without reconstructing data.
- Live vehicle tracking and service alerts were isolated from the routing service and remained available.
- Heap diagnostics allowed the team to identify the problematic station hierarchy without reproducing the full outage locally.

## What Did Not Go Well

- The investigation initially focused on application code rather than the newly activated dataset.
- Automated worker restarts worsened dependency saturation.
- There was no one-step rollback for schedule data and derived caches.
- Customer-facing communication was delayed.
- Third-party API consumers received generic server errors without a service-degradation indicator or useful retry guidance.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add cycle detection and hierarchy-depth validation to the schedule import pipeline. | Transit Data Engineering | P0 | August 7, 2026 |
| Add visited-node tracking and a traversal limit to transfer-graph construction. | Routing Platform | P0 | August 7, 2026 |
| Introduce canary activation for schedule datasets before fleet-wide rollout. | Platform Engineering | P0 | August 21, 2026 |
| Limit graph rebuild concurrency by default and add randomized restart backoff. | Site Reliability Engineering | P0 | August 14, 2026 |
| Build an automated rollback that restores the prior dataset, invalidates derived caches, and safely restarts workers. | Platform Engineering | P1 | September 4, 2026 |
| Add dataset-version labels to latency, memory, and error-rate telemetry. | Observability Team | P1 | August 28, 2026 |
| Alert on repeated memory-limit restarts tied to the same dataset version. | Site Reliability Engineering | P1 | August 14, 2026 |
| Load-test graph construction using malformed, cyclic, and unusually dense transfer data. | Quality Engineering | P1 | September 11, 2026 |
| Automatically publish a status-page incident when trip-planning availability remains below 95% for five minutes. | Customer Communications and SRE | P1 | August 28, 2026 |
| Return a documented `Retry-After` header during planner-wide degradation. | API Team | P2 | September 18, 2026 |
| Update the incident runbook with schedule rollback and cache-recovery procedures. | Site Reliability Engineering | P1 | August 7, 2026 |
| Conduct a recovery exercise using a deliberately invalid schedule dataset. | Reliability Program | P2 | October 2, 2026 |

## Lessons Learned

Schedule data is executable input to the routing platform and must receive the same safeguards as application code. Structural validation alone was insufficient because the failure depended on relationships between otherwise valid records.

Future schedule releases will be treated as production changes: validated for behavioral invariants, exercised against a canary, monitored by dataset version, and automatically rolled back when health checks fail.