# Postmortem: Journey Planning Outage During Morning Commute

**Incident date:** July 22, 2026  
**Duration:** 72 minutes  
**Severity:** SEV-1  
**Affected services:** Web journey planner, mobile applications, public trip-planning API  
**Status:** Resolved

## Summary

On July 22, 2026, the municipal transit journey planner was unavailable or returned incomplete results for 72 minutes during the morning commute. A scheduled transit-data import introduced an unusually large service update. The import caused sustained lock contention in the routing database, while an automatic retry mechanism created additional load and exhausted the API connection pool.

Static schedules and service-alert pages remained available. Real-time arrivals continued to display for previously selected stops, but most customers could not search for new trips.

## Customer Impact

Between 07:18 and 08:30 local time:

- Approximately 68% of journey-planning requests failed.
- Successful requests had a p95 response time of 24 seconds, compared with the normal 1.8 seconds.
- Mobile applications displayed “Unable to plan trip” or timed out.
- The public API returned HTTP 503 responses to third-party accessibility and mobility applications.
- Some successful results omitted two temporary shuttle routes added that morning.
- Call-centre trip-planning volume increased by 41%.

We estimate that 46,000 customers encountered at least one failed request. No payment, account, or customer-location data was lost or exposed.

## Detection

The incident was detected by API error-rate alerts at 07:21, three minutes after customer impact began. The on-call engineer acknowledged the alert at 07:24.

Our synthetic journey-planning checks did not alert first because they reused cached results for a single route. This delayed confirmation that uncached searches were broadly affected.

## Timeline

All times are local.

| Time | Event |
|---|---|
| 06:55 | The scheduled transit-data import begins. |
| 07:12 | Validation completes and the importer starts publishing the new dataset. |
| 07:18 | Database lock duration increases sharply. Uncached journey-planning requests begin timing out. |
| 07:21 | API error-rate alert fires. |
| 07:24 | The on-call engineer acknowledges the alert and begins investigation. |
| 07:28 | The connection pool reaches its maximum size. API instances begin returning HTTP 503 responses. |
| 07:33 | The incident is declared SEV-1. Transit operations and customer communications teams are notified. |
| 07:39 | Engineers identify repeated import retries and stop the importer. Database load remains elevated because queued transactions continue running. |
| 07:46 | A public service notice is posted to the website and social channels. |
| 07:51 | Engineers terminate obsolete import transactions and restart API instances in controlled batches. |
| 08:02 | Error rates begin declining. Cached and simple trip searches recover first. |
| 08:14 | The previous validated transit dataset is restored. |
| 08:24 | Journey-planning success rate returns to normal. |
| 08:30 | Real-time feeds and third-party API traffic are fully restored. The incident is marked resolved. |
| 10:05 | The new transit dataset is republished using the manual low-impact procedure. |

## Root Cause

The immediate cause was lock contention created while publishing a large transit-data update to the routing database.

The publishing process updated route, stop, trip, and calendar tables within a single transaction. This normally completes in under 40 seconds. The July 22 dataset included temporary shuttle service and a citywide schedule revision, increasing the transaction to approximately 18 times its typical size.

After 90 seconds, the importer incorrectly classified the slow transaction as failed and started another attempt without confirming that the original database transaction had ended. Three overlapping attempts competed for the same tables and blocked read queries used by the journey-planning API.

As request latency increased, API instances opened additional database connections until the shared connection pool was exhausted. At that point, even requests that did not require the locked records began failing.

## Contributing Factors

- The importer used in-place updates rather than building a new dataset and switching versions atomically.
- Retry logic was based on client-side timeouts and did not verify transaction state before retrying.
- The staging environment contained a smaller dataset and did not reproduce production lock duration.
- Database connection limits were shared between the importer and customer-facing APIs.
- Synthetic monitoring tested a cacheable route and did not exercise uncached or cross-region searches.
- The runbook documented how to stop the importer but not how to identify and terminate orphaned transactions.
- The initial service notice was delayed because the communications team lacked a pre-approved journey-planner outage template.

## Resolution and Recovery

Engineers stopped the importer, terminated its obsolete database transactions, and restarted API instances in batches to clear exhausted connections without causing a new traffic spike.

Because the new dataset had not published consistently, the team restored the previous validated version. Once service stabilized, the new dataset was imported using a manual blue-green process and validated before activation.

## What Went Well

- Error-rate monitoring detected the incident within three minutes.
- The on-call engineer escalated quickly after confirming database contention.
- The previous transit dataset remained available and was restored without data loss.
- Coordination between engineering, transit operations, and the call centre reduced conflicting customer guidance.
- Real-time vehicle-location ingestion continued operating throughout the incident.

## What Went Poorly

- Automated retries amplified the original database contention.
- Monitoring did not represent typical uncached customer searches.
- The customer-facing status update was published 28 minutes after impact began.
- The incident runbook lacked commands and ownership guidance for orphaned import transactions.
- Third-party API consumers received generic 503 responses without retry guidance.

## Action Items

| Action | Owner | Priority | Target date |
|---|---|---:|---|
| Replace in-place dataset publication with versioned tables and an atomic active-version switch. | Routing Platform | P0 | August 21, 2026 |
| Disable automatic import retries until transaction-state checks are implemented. | Data Pipeline | P0 | Completed |
| Add idempotency keys and server-side cancellation to the import workflow. | Data Pipeline | P0 | August 14, 2026 |
| Isolate importer and customer API database connection pools. | Database Engineering | P0 | August 7, 2026 |
| Add load testing with production-scale schedule and temporary-service datasets. | Performance Engineering | P1 | August 28, 2026 |
| Expand synthetic monitoring to cover uncached, accessible, cross-region, and multimodal trips. | Reliability Engineering | P1 | August 12, 2026 |
| Alert on long-running import transactions and database lock wait time. | Database Engineering | P1 | August 5, 2026 |
| Add circuit breaking so degraded imports cannot exhaust API connections. | Routing Platform | P1 | September 4, 2026 |
| Update the incident runbook with orphaned-transaction recovery procedures. | Reliability Engineering | P1 | Completed |
| Create pre-approved service-notice templates for journey-planning outages. | Customer Communications | P2 | August 7, 2026 |
| Return structured retry information in public API error responses. | Developer Platform | P2 | September 11, 2026 |

## Lessons Learned

Batch data publication must be treated as part of the customer-facing request path, even when it runs as a scheduled background process. A valid but unusually large dataset should degrade publication speed, not journey-planning availability.

Future transit-data releases will be built and validated separately from the active routing dataset. Customer traffic will continue using the current version until the replacement is ready for an atomic switch.