# Incident Postmortem: Journey Planning Service Outage

**Incident date:** July 17, 2026  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-1  
**Services affected:** Web trip planner, mobile applications, public journey-planning API  
**Status:** Resolved

## Summary

On July 17, 2026, the municipal transit journey-planning service experienced a major outage during the morning commute. Riders could not reliably search for routes, departure times, or accessible-trip options. Most requests returned HTTP 503 errors; a smaller number timed out or displayed incomplete itineraries.

The incident began after a routine deployment introduced a database query that bypassed the application’s existing index for stop-time lookups. Elevated query latency exhausted the API connection pool. Automatic retries then increased database load, preventing the service from recovering without intervention.

Service was restored by rolling back the deployment, disabling client retries at the gateway, and restarting the affected API instances in controlled batches.

## Customer Impact

Between 07:18 and 09:05 local time:

- Approximately 81% of journey-planning requests failed.
- An estimated 46,000 riders encountered at least one error.
- The web planner and mobile applications displayed “Unable to plan trip” or continued loading until timeout.
- Third-party accessibility and commuter applications using the public API received elevated 503 responses.
- Real-time vehicle-location feeds remained available, but most customers could not use them to generate complete itineraries.
- Transit operations, fare payment, vehicle dispatch, and station information displays were not affected.

The outage occurred during the weekday morning peak and disproportionately affected riders planning unfamiliar, multimodal, or wheelchair-accessible trips.

## Timeline

All times are local.

| Time | Event |
|---|---|
| 06:52 | Version `planner-api-2026.07.17.1` deployed to 10% of production instances. Automated health checks passed. |
| 07:04 | Deployment expanded to 100% after error rate and CPU remained within release thresholds. |
| 07:12 | Database query latency began increasing as commuter traffic rose. |
| 07:18 | API connection-pool utilization reached 100%; journey-planning errors exceeded 20%. Incident began. |
| 07:21 | Automated alert fired for elevated HTTP 503 responses. |
| 07:25 | On-call engineer acknowledged the alert and began investigating API saturation. |
| 07:31 | Error rate exceeded 75%. Incident declared SEV-1; database and platform responders paged. |
| 07:38 | Responders identified a sharp increase in stop-time query duration and database connections. |
| 07:44 | Initial attempt to scale API instances increased connection pressure and worsened database latency. Scaling was stopped. |
| 07:51 | Communications team published a service advisory recommending official route maps and stop displays. |
| 08:03 | Engineers traced the slow query to the morning deployment. Rollback initiated. |
| 08:14 | Rollback completed, but error rates remained elevated because queued requests and automatic retries continued consuming connections. |
| 08:22 | Gateway retries disabled for journey-planning requests. |
| 08:30 | API instances restarted in batches to clear saturated pools and stale request queues. |
| 08:42 | Successful request rate recovered above 80%. |
| 08:53 | Error rate returned below 2%; latency continued improving. |
| 09:05 | Metrics remained stable for ten minutes. Incident resolved. |
| 09:32 | Public service advisory cleared. |

## Root Cause

The deployment changed the accessible-trip query to support filtering by temporary elevator outages. The new query applied a timezone conversion function directly to the indexed `departure_time` column:

```sql
WHERE convert_timezone(service_timezone, departure_time) BETWEEN ? AND ?
```

Applying the function to the column prevented the database optimizer from using the existing composite index on route, service date, and departure time. At low traffic, the resulting scans were slow but remained below the API timeout. During the morning peak, concurrent scans increased database I/O and extended query duration from a median of 42 milliseconds to more than 11 seconds.

Each API instance had a fixed pool of 40 database connections. Slow queries occupied all available connections, causing new requests to wait and eventually fail. The API gateway retried eligible failed requests twice, and the mobile clients retried some timeouts independently. This retry amplification created roughly 2.6 database attempts for each rider request at the incident peak.

Rolling back the code stopped new inefficient queries, but the accumulated request backlog and retry traffic kept connection pools saturated until retries were disabled and instances were restarted.

## Contributing Factors

- Performance tests used a dataset containing approximately 12% of production stop-time records.
- Release validation measured endpoint latency but did not inspect database query plans.
- The canary deployment occurred before peak demand, when traffic was insufficient to expose connection-pool exhaustion.
- The deployment controller promoted the release based on CPU and error rate, without evaluating database query latency or pool utilization.
- Gateway and mobile-client retries were configured independently and lacked a shared retry budget.
- Scaling the API tier automatically increased the number of possible database connections, adding pressure to the constrained dependency.
- The accessible-trip endpoint shared the same connection pool as standard journey-planning requests, increasing the outage’s scope.
- The rollback procedure did not include clearing queued requests or recycling saturated connection pools.

## Detection

The incident was detected by an HTTP error-rate alert three minutes after customer-visible failures began. Database latency and connection-pool saturation dashboards showed the underlying condition, but neither metric had a paging alert.

The initial health checks were insufficient because they used simple planning requests against recently cached routes. They did not exercise accessible-trip filtering or representative peak-hour time ranges.

## Resolution and Recovery

Engineers restored service by:

1. Rolling back the release containing the inefficient query.
2. Disabling automatic gateway retries for journey-planning requests.
3. Restarting API instances in controlled batches to clear queued work and saturated connection pools.
4. Temporarily reducing request timeouts to prevent new backlogs.
5. Monitoring database latency, connection usage, and successful journey plans until the service remained stable.

The query change was subsequently rewritten to calculate UTC range boundaries before execution, allowing the database to use the existing index.

## What Went Well

- The on-call engineer acknowledged the initial alert within four minutes.
- Application, platform, database, and communications responders joined the incident quickly.
- The deployment was traceable to a single release and could be rolled back without a data migration.
- Real-time vehicle feeds and operational transit systems were isolated from the planner database.
- Public updates were published during the incident and cleared after recovery.

## What Went Poorly

- The release passed automated tests despite containing a query that could not scale to production traffic.
- The canary process did not generate representative load.
- Scaling the API tier worsened the incident before database saturation was fully understood.
- Multiple retry layers amplified demand during dependency failure.
- Responders lacked a documented procedure for draining queued requests after rollback.
- The status advisory did not initially mention that third-party applications were also affected.

## Corrective and Preventive Actions

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add production-scale stop-time data to performance-test environments. | Data Platform | P0 | August 14, 2026 |
| Require query-plan regression checks for changes to journey-planning SQL. | Planner Engineering | P0 | August 21, 2026 |
| Alert on database connection-pool saturation and sustained query latency. | Site Reliability Engineering | P0 | August 7, 2026 |
| Implement a shared retry budget with exponential backoff and jitter across gateway and clients. | Platform Engineering | P0 | September 4, 2026 |
| Add database health and pool utilization to deployment promotion criteria. | Release Engineering | P1 | August 28, 2026 |
| Run canary synthetic traffic using representative peak-hour routes and accessibility filters. | Quality Engineering | P1 | August 28, 2026 |
| Cap aggregate database connections independently of API instance count. | Database Engineering | P1 | September 11, 2026 |
| Isolate expensive accessibility queries with a separate pool and concurrency limit. | Planner Engineering | P1 | September 18, 2026 |
| Update rollback procedures to include queue draining and controlled instance recycling. | Site Reliability Engineering | P1 | August 14, 2026 |
| Add load-shedding that preserves simple direct-route and next-departure requests during saturation. | Planner Engineering | P2 | October 2, 2026 |
| Update incident communication templates to cover public API consumers and accessible alternatives. | Customer Communications | P2 | August 21, 2026 |
| Conduct a peak-traffic database failure exercise. | Reliability Program | P2 | September 25, 2026 |

## Lessons Learned

A healthy canary at low traffic did not demonstrate that the release was safe under peak load. Journey-planning performance depends heavily on database access patterns, so endpoint-level checks alone are insufficient.

Retries also need to be treated as a finite system-wide resource. Independent retry behavior at the gateway and client layers converted a localized performance regression into sustained overload and delayed recovery.

Future releases affecting timetable or accessibility queries will require production-scale performance validation, query-plan review, and peak-load canary traffic before full deployment.