# Postmortem: Trip Planning and Real-Time Arrival Outage

**Incident date:** July 17, 2026  
**Duration:** 08:07–09:41 EDT  
**Severity:** SEV-1  
**Services affected:** Web trip planner, mobile trip planner, real-time arrival API  
**Status:** Resolved

## Summary

On July 17, the municipal transit trip-planning platform experienced a 94-minute outage during the morning commute. Customers were unable to plan new trips, and real-time arrival information was unavailable or stale across the website, mobile applications, and third-party integrations.

The incident began after a routine deployment introduced an inefficient database query in the real-time vehicle ingestion service. Morning peak traffic caused the query volume to exceed the primary database's capacity. Connection pools filled, requests queued, and dependent trip-planning services began timing out.

Service was restored by rolling back the deployment, restarting saturated application instances, and temporarily reducing the frequency of real-time vehicle updates.

## Customer Impact

Between 08:07 and 09:41 EDT:

- Approximately 184,000 trip-planning requests failed or timed out.
- 72% of web and mobile trip-planning sessions encountered at least one error.
- Real-time arrivals were unavailable for most routes.
- Previously cached arrival estimates remained visible for up to 18 minutes without a clear stale-data warning.
- Third-party applications using the public API received elevated HTTP 500 and 503 responses.
- Static schedules and service-alert pages remained available.
- Transit operations, fare payment, and vehicle dispatch systems were not affected.

The outage occurred during the weekday morning peak and disproportionately affected customers making transfers or responding to active service disruptions.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 07:42 | Version `realtime-ingest-2026.07.17.1` deployed to production after automated tests and canary checks passed. |
| 07:55 | Morning vehicle-update volume began increasing as expected. Database query latency rose gradually but remained below the alert threshold. |
| 08:07 | Primary database connection utilization reached 100%. Trip-planning API latency increased sharply. |
| 08:09 | Automated alert triggered for elevated API error rates. The on-call engineer began investigating. |
| 08:13 | Mobile and web clients began showing widespread “Unable to plan trip” errors. |
| 08:17 | Incident declared SEV-1. Database and trip-planning teams were paged. |
| 08:23 | Engineers identified connection-pool exhaustion but initially suspected an external real-time feed backlog. |
| 08:31 | The team paused one vehicle-data feed. Database load decreased briefly, then returned to saturation. |
| 08:38 | Incident commander correlated the degradation with the 07:42 deployment. |
| 08:44 | Rollback initiated. |
| 08:51 | Rollback completed, but several application instances retained exhausted connection pools. |
| 08:57 | Saturated trip-planning and ingestion instances restarted in controlled batches. |
| 09:08 | Database latency began returning to normal. Successful trip-planning requests reached 70%. |
| 09:16 | Real-time update frequency temporarily reduced from every 10 seconds to every 30 seconds. |
| 09:27 | Error rates returned below the alert threshold. Real-time arrivals began updating consistently. |
| 09:41 | Full service confirmed across web, mobile, and public API clients. Incident moved to monitoring. |
| 11:20 | Normal 10-second real-time update frequency restored after additional validation. |

## Root Cause

The deployment changed vehicle-position processing from single-route updates to batched regional updates. The new implementation queried stop mappings using a list of route identifiers, but the query did not use the existing composite index because the route identifiers were cast to a different database type.

As a result, each ingestion batch performed a sequential scan of the stop-mapping table. This behavior was not visible during canary testing because the canary received only 2% of production traffic and operated before peak vehicle-update volume.

During the morning commute, overlapping scans consumed database CPU and held connections substantially longer than normal. The ingestion service exhausted its connection pool, followed by the shared database connection limit. The trip-planning API could no longer obtain connections for routing and arrival requests, causing widespread timeouts and HTTP 500 responses.

## Contributing Factors

- The ingestion and trip-planning services shared the same database connection capacity without reserved limits.
- Performance tests used a reduced stop-mapping dataset and did not reproduce production query plans.
- Deployment checks monitored application error rates but did not compare database query plans or rows scanned.
- The database CPU alert threshold required five consecutive minutes above 90%, delaying detection during the initial ramp-up.
- Cached real-time data lacked a prominent stale-data indicator.
- The rollback procedure did not automatically recycle application instances with exhausted connection pools.
- The initial investigation focused on upstream feed health because similar symptoms had previously been caused by feed backlogs.

## Resolution and Recovery

The team rolled back the ingestion deployment, restarted affected application instances, and temporarily reduced the real-time update frequency. These actions released blocked connections and lowered database demand enough for queued requests to clear.

After service recovered, the team corrected the query parameter types, added the required index coverage, and validated the execution plan against a production-sized dataset before redeployment.

## What Went Well

- API error-rate monitoring detected the incident within two minutes of widespread customer impact.
- The incident command process was established within ten minutes.
- Static schedules and service alerts remained available throughout the outage.
- The previous application version was deployable without a database rollback.
- Cross-team collaboration allowed the deployment correlation to be identified quickly after the initial feed investigation.

## What Went Poorly

- Canary traffic was too low and too early to expose peak-load database behavior.
- Database saturation affected multiple customer-facing services because connection capacity was shared.
- Customers could not reliably distinguish stale arrival estimates from current data.
- The rollback restored code but did not fully restore service until instances were restarted.
- The status page was not updated until 08:34, 27 minutes after customer impact began.

## Action Items

| Action | Owner | Priority | Target date |
|---|---|---:|---|
| Add production-scale query-plan validation to the ingestion deployment pipeline. | Data Platform | P0 | August 7, 2026 |
| Separate ingestion and trip-planning database connection pools with enforced per-service limits. | Platform Engineering | P0 | August 14, 2026 |
| Add automated rollback criteria based on database latency, connection utilization, and rows scanned. | Site Reliability Engineering | P0 | August 21, 2026 |
| Create a peak-volume load test using a production-sized anonymized stop-mapping dataset. | Performance Engineering | P1 | August 28, 2026 |
| Update canary strategy to include representative traffic and a peak-load soak period. | Release Engineering | P1 | August 21, 2026 |
| Display a clear warning when real-time arrivals are older than two minutes. | Rider Applications | P1 | September 4, 2026 |
| Make application instance recycling part of the ingestion rollback procedure. | Platform Engineering | P1 | August 7, 2026 |
| Lower database saturation alert latency and add alerts for connection wait time. | Site Reliability Engineering | P1 | August 7, 2026 |
| Add a runbook decision point for correlating incidents with recent deployments. | Site Reliability Engineering | P2 | August 14, 2026 |
| Automate status-page updates when a SEV-1 customer-facing incident is declared. | Service Operations | P2 | September 11, 2026 |

## Lessons Learned

A successful low-traffic canary does not validate database behavior under peak concurrency, particularly when a code change affects query construction. Future releases involving shared data stores must be evaluated using production-scale data, representative traffic, and explicit database safety signals.

We also need stronger isolation between real-time ingestion and customer-facing trip planning. A degradation in one workload should reduce data freshness gracefully rather than exhaust the resources required to serve all trip-planning requests.