# Postmortem: Transit Journey Planning Outage — June 18, 2026

**Incident date:** June 18, 2026  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-1  
**Affected systems:** Web trip planner, mobile applications, public journey-planning API  
**Status:** Resolved

## Summary

From 07:42 to 09:29 local time, customers experienced failed or incomplete trip-planning requests across the municipal transit website, mobile applications, and public API. Vehicle tracking and service-alert systems remained operational, but most customers could not generate new itineraries.

The outage occurred during the morning commute and followed a routine deployment to the routing service. A database migration introduced a new index on the stop-time table. The migration was expected to run concurrently, but a deployment configuration error caused it to acquire an exclusive table lock. Routing workers waiting on the locked table exhausted the database connection pool, and automatic retries amplified the load.

We restored service by terminating the migration, restarting routing workers in controlled batches, and temporarily reducing retry attempts. No customer or operational data was lost.

## Customer Impact

During the incident:

- 81% of journey-planning requests failed or exceeded the 10-second client timeout.
- Approximately 126,000 trip-planning requests were affected.
- Web and mobile customers received generic “Unable to plan trip” messages.
- Some successful itineraries omitted real-time arrival adjustments and displayed scheduled times only.
- Third-party accessibility and mobility applications using the public API experienced elevated timeouts and HTTP 503 responses.
- Customer support received 438 incident-related contacts.
- Vehicle location feeds, fare payment, operator communications, and service-alert publishing were not affected.

The outage disproportionately affected customers planning transfers and accessible journeys because those trips generally require more routing queries. Previously loaded itineraries remained available in the mobile applications, although their real-time arrival estimates stopped updating in some cases.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 07:31 | Routing service version `2026.06.18-1` begins deployment after passing automated tests and staging validation. |
| 07:38 | Application deployment completes. The associated database migration starts automatically. |
| 07:42 | Database query latency rises sharply as the migration acquires an exclusive lock on the `stop_times` table. |
| 07:44 | Routing workers begin exhausting database connections. Failed requests trigger automatic retries. |
| 07:46 | The first latency alert fires for the internal routing API. |
| 07:49 | On-call engineer acknowledges the alert and begins investigation. |
| 07:53 | Web and mobile error-rate alerts fire. Public API availability falls below the service-level objective. |
| 07:57 | Incident declared SEV-1. Transit operations, customer support, and communications teams are notified. |
| 08:03 | Engineers identify database connection exhaustion but initially attribute it to increased commuter demand. |
| 08:11 | Application instances are scaled from 24 to 40. This increases database connection pressure and worsens timeouts. |
| 08:18 | Database team joins the incident and identifies the blocked queries and long-running migration. |
| 08:24 | Deployment is halted. The application release is rolled back, but the database migration continues running. |
| 08:31 | Engineers confirm that the migration is holding an exclusive lock rather than creating the index concurrently. |
| 08:37 | The migration is terminated. Database lock contention begins to clear. |
| 08:43 | Retry attempts are reduced from three to one, and worker restarts begin in batches. |
| 08:52 | Successful request volume starts recovering; error rate remains elevated due to a backlog of queued requests. |
| 09:04 | Public API availability returns above 95%. |
| 09:16 | Mobile and web success rates return to normal levels. Real-time enrichment remains partially disabled. |
| 09:29 | Real-time enrichment is restored. Incident marked resolved. |
| 10:15 | Customer-facing incident notice updated with resolution details. |
| 13:00 | Data integrity checks complete; no data loss or corruption is found. |

## Root Cause

The immediate cause was an exclusive database lock acquired while creating an index on the `stop_times` table, which is read by nearly every journey-planning request.

The migration was written to use PostgreSQL’s concurrent index creation mode. However, the deployment system wrapped all migration statements in a transaction by default. Because concurrent index creation cannot run inside a transaction, an environment-specific migration helper replaced the statement with a standard index creation operation. That fallback behavior was intended for local development but was enabled in production through a shared configuration default.

The resulting operation locked the table against reads for longer than expected. Routing requests accumulated while waiting for the lock, consuming all available database connections. The routing service retried timed-out queries three times without sufficient backoff, multiplying demand. Scaling the application added more workers and database connections, further increasing contention and delaying recovery.

## Contributing Factors

- The staging database contained substantially less stop-time data, so the same migration completed in under five seconds without causing visible impact.
- Migration tests verified schema correctness but did not assert the lock behavior of production-bound statements.
- Database migrations ran automatically as part of the application deployment and did not require a separate approval for high-traffic tables.
- The fallback from concurrent to standard index creation emitted a warning but did not fail the deployment.
- Alerts detected latency and errors but did not immediately identify database lock contention.
- The operational runbook recommended horizontal scaling for elevated routing latency without first checking database saturation.
- Retry behavior used fixed delays and did not enforce a total retry budget.
- The routing service and real-time enrichment service shared the same database connection pool, allowing routing contention to degrade real-time updates.

## Detection and Response

Automated monitoring detected the incident four minutes after customer impact began. The on-call engineer responded within three minutes, but diagnosis took longer because initial dashboards emphasized CPU, request volume, and connection-pool usage rather than active database locks.

The first mitigation—adding application capacity—was ineffective and increased pressure on the database. Recovery began after the database migration was terminated. Controlled worker restarts and reduced retry attempts cleared the backlog without producing another connection surge.

Communications teams published an initial service notice 28 minutes after impact began. That notice correctly stated that trip planning was unavailable, but it did not clarify that vehicle tracking, fare payment, and transit operations remained functional.

## Resolution

Engineers terminated the index migration, rolled back the application deployment, reduced routing retries, and restarted workers in batches. Once normal query latency returned, real-time enrichment was gradually re-enabled.

The index was created later during a scheduled maintenance window using a separately reviewed, non-transactional concurrent migration. Engineers monitored table locks and replica lag throughout that operation.

## What Went Well

- Automated monitoring detected the customer-facing failure within minutes.
- On-call, database, operations, support, and communications teams assembled quickly after the SEV-1 declaration.
- Existing controls prevented data corruption when the migration was terminated.
- Batch restarts and gradual feature restoration avoided a second traffic surge.
- Vehicle operations and fare systems were isolated from the journey-planning outage.

## What Went Poorly

- Production migration behavior differed from the reviewed migration source.
- Staging data volume and query concurrency did not adequately represent production.
- The initial mitigation increased database pressure.
- Retry amplification significantly expanded the outage.
- Lock diagnostics were not visible in the primary incident dashboard.
- Customer communications were slower and less specific than required for a morning commute incident.

## Action Items

| Action | Owner | Priority | Target Date |
|---|---|---:|---|
| Remove the production fallback from concurrent to standard index creation; fail migrations that cannot preserve the requested lock behavior. | Database Platform | P0 | June 25, 2026 |
| Separate database migrations from application deployments and require explicit approval for changes to high-traffic tables. | Release Engineering | P0 | July 2, 2026 |
| Add CI checks that inspect migration statements, transaction mode, and expected lock level. | Developer Productivity | P0 | July 9, 2026 |
| Add exponential backoff, jitter, and a total retry budget to routing database calls. | Journey Planning | P0 | July 2, 2026 |
| Cap routing worker database connections independently of application replica count. | Journey Planning | P0 | June 27, 2026 |
| Add active-lock, blocked-query, and oldest-transaction panels to the primary incident dashboard. | Site Reliability Engineering | P1 | June 30, 2026 |
| Alert when a production migration blocks customer queries for more than 15 seconds. | Database Platform | P1 | June 30, 2026 |
| Update the routing latency runbook to require database saturation and lock checks before scaling. | Site Reliability Engineering | P1 | June 23, 2026 |
| Create a production-scale migration test environment using anonymized traffic and data distributions. | Quality Engineering | P1 | July 31, 2026 |
| Separate routing and real-time enrichment database connection pools. | Transit Data Platform | P1 | July 16, 2026 |
| Add graceful degradation that returns schedule-only itineraries when real-time enrichment is unavailable. | Journey Planning | P1 | July 23, 2026 |
| Update incident communication templates to distinguish trip-planning failures from operational transit disruptions. | Customer Communications | P2 | June 26, 2026 |
| Run a cross-team game day covering database lock contention and retry amplification. | Incident Management | P2 | August 7, 2026 |

## Lessons Learned

Database migration safety depends on the exact statement executed in production, not only the migration code reviewed by engineers. Any system that rewrites migration behavior must fail closed when it cannot preserve the requested safety properties.

The incident also demonstrated that retries and horizontal scaling can turn a constrained dependency into a broader outage. Capacity controls, bounded retries, and dependency-aware runbooks are necessary to keep routine mitigation steps from increasing load during partial failures.