# Postmortem: Transit Planner Routing Outage — July 22, 2026

**Incident date:** July 22, 2026  
**Duration:** 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Affected services:** Web trip planner, mobile applications, public journey-planning API

## Summary

On July 22, 2026, the municipal transit planner returned errors or incomplete itineraries for 47 minutes during the morning commute. A routine schedule-data deployment caused excessive cache invalidations, sharply increasing requests to the routing cluster. An autoscaling configuration error prevented additional routing workers from starting, and the existing workers became saturated.

Service was restored by rolling back the schedule deployment, increasing routing-cluster capacity manually, and warming the itinerary cache. No customer or payment data was lost.

## Customer Impact

Between 07:41 and 08:28 local time:

- Approximately 68% of trip-planning requests failed with HTTP 503 errors or timed out.
- About 21% returned incomplete itineraries that omitted one or more bus routes.
- Real-time departure boards and vehicle-location maps remained available.
- Fare payment, account management, and operator dispatch systems were unaffected.
- Third-party applications using the public API experienced the same routing failures.
- An estimated 94,000 trip-planning requests were affected.

Customers traveling during the morning commute may have been unable to find routes, received results limited to rail services, or seen repeated “Try again” messages.

## Timeline

All times are local.

| Time | Event |
|---|---|
| 07:35 | The automated schedule-data pipeline began deploying the next service-day dataset. |
| 07:39 | Deployment completed and published a cache-invalidation event for updated stops, routes, and trips. |
| 07:41 | Routing latency crossed the alert threshold; intermittent HTTP 503 responses began. |
| 07:43 | The on-call engineer was paged for elevated error rate and routing-worker saturation. |
| 07:47 | Customer support reported a spike in complaints from web and mobile users. |
| 07:49 | The incident was declared SEV-1, and an incident commander was assigned. |
| 07:53 | Engineers confirmed that routing workers were at maximum CPU and request queues were growing. |
| 07:57 | The team observed that the routing autoscaler was requesting additional capacity, but no new workers were becoming ready. |
| 08:02 | Investigation identified an invalid node-pool selector introduced during an infrastructure change the previous evening. |
| 08:07 | The schedule-data deployment was rolled back to reduce cache churn. |
| 08:11 | Engineers corrected the node-pool selector and manually increased the routing-worker count. |
| 08:17 | New workers became ready; error rates began declining. |
| 08:21 | The most common itinerary queries were pre-warmed in the cache. |
| 08:28 | Error rate and latency returned to normal; service was declared restored. |
| 09:15 | Monitoring confirmed stable operation through the remainder of the morning peak. |

## Detection

Automated monitoring detected elevated routing latency two minutes after the outage began and paged the on-call engineer. The initial alert identified saturation in the routing service but did not indicate that autoscaled workers were unschedulable.

Customer support independently reported the issue six minutes after the first alert. The public status page was updated at 08:00, 19 minutes after customer impact began.

## Root Cause

The immediate trigger was a schedule-data deployment containing routine timetable changes across several bus routes. The deployment system invalidated itinerary-cache entries using a dataset-wide version key. As a result, even unchanged routes lost their cached itineraries.

The resulting cache miss rate increased from 14% to 93%. Each miss required a full pathfinding calculation, increasing CPU demand on the routing cluster by approximately six times.

The cluster was designed to absorb this load through horizontal autoscaling. However, an infrastructure change made the previous evening had updated the labels on the dedicated routing node pool without updating the routing workers’ node selector. Existing workers continued running, but newly requested workers could not be scheduled. The autoscaler therefore increased the desired replica count without adding usable capacity.

The combination of broad cache invalidation and unavailable autoscaling exhausted the existing routing workers and caused request queues, timeouts, and HTTP 503 responses.

## Contributing Factors

- The schedule-data deployment invalidated cache entries at the dataset level rather than only for changed routes and stops.
- The infrastructure change was validated against existing workloads but did not include a scale-from-minimum test.
- Monitoring alerted on worker saturation but not on unschedulable routing pods.
- Load testing covered normal schedule imports but did not simulate a full cache invalidation during peak demand.
- Mobile clients retried failed requests immediately, adding approximately 18% more traffic during the incident.
- The rollback procedure restored the previous dataset but did not automatically repopulate high-traffic cache entries.
- The status-page update required manual approval, delaying public communication.

## Resolution and Recovery

The incident team rolled back the schedule dataset, corrected the routing workers’ node selector, and manually increased cluster capacity. Once new workers were available, request queues drained and error rates declined.

Engineers then pre-warmed cache entries for common origin-destination pairs, which accelerated recovery. Service was considered restored after routing latency, error rate, queue depth, and cache hit rate remained within normal thresholds for ten minutes.

## What Went Well

- Automated monitoring detected the issue before most customer reports arrived.
- The incident commander established clear ownership of mitigation, investigation, and communications.
- The schedule deployment was reversible and rolled back without data corruption.
- Routing capacity recovered quickly after the scheduling constraint was corrected.
- Real-time vehicle data and fare-payment systems were isolated from the routing failure.

## What Went Poorly

- Deployment and infrastructure changes were assessed independently, so their combined failure mode was not identified.
- The autoscaler appeared healthy because desired replica counts increased even though workers were not schedulable.
- The cache invalidation strategy amplified a small schedule update into a full-cluster load event.
- Immediate client retries increased load while the service was already saturated.
- Public communication lagged behind internal incident declaration.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Replace dataset-wide cache invalidation with route- and stop-level invalidation. | Routing Platform | P0 | August 14, 2026 |
| Add deployment validation that confirms routing workers can scale from minimum to peak capacity. | Infrastructure | P0 | August 7, 2026 |
| Alert when routing pods remain unschedulable for more than two minutes. | Observability | P0 | August 5, 2026 |
| Add a canary stage that measures cache-miss rate before schedule data is published globally. | Transit Data | P1 | August 21, 2026 |
| Run peak-load tests with a cold itinerary cache before each major schedule change. | Performance Engineering | P1 | August 28, 2026 |
| Add exponential backoff and jitter to web and mobile trip-planning retries. | Client Applications | P1 | September 4, 2026 |
| Automate pre-warming of high-volume itinerary queries after deployments and rollbacks. | Routing Platform | P1 | August 21, 2026 |
| Prevent node-label changes when active workload selectors would become invalid. | Infrastructure | P1 | August 14, 2026 |
| Allow the incident commander to publish an initial status-page notice without secondary approval. | Service Operations | P2 | August 11, 2026 |
| Conduct a resilience exercise covering routing saturation during the morning peak. | Reliability Engineering | P2 | September 18, 2026 |

## Lessons Learned

Autoscaling is only effective when new capacity can actually be scheduled, and desired replica count alone is not a sufficient health signal. Schedule-data deployments must also be treated as production load events because cache behavior can make a logically small update operationally large.

Future changes will be evaluated across the full request path—data publication, cache invalidation, routing capacity, client retries, and customer communication—rather than as isolated components.