# Postmortem: Station Unlock and Trip-Start Outage

**Incident date:** July 8, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Affected region:** Greater Lakeshore service area  
**Status:** Resolved

## Summary

On July 8, 2026, riders across the Greater Lakeshore bike-sharing network were intermittently unable to unlock bicycles or begin trips. The incident began after a routine deployment introduced an incompatible cache-key format between the Trip API and the station authorization service.

Requests for stations with recently refreshed metadata bypassed the cache and reached the authorization database directly. During the evening commute, the resulting database load exhausted the connection pool, causing elevated latency and widespread request failures. Service was restored by rolling back the deployment, restarting saturated application instances, and temporarily increasing database connection capacity.

## Customer Impact

From 16:42 to 18:59 EDT:

- Approximately 31,400 unlock attempts failed or timed out.
- An estimated 12,800 riders were unable to start a trip on their first attempt.
- 46% of trip-start requests failed during the peak of the incident.
- Mobile applications frequently displayed “Bike unavailable” or a generic network error, even when bicycles were physically available.
- Kiosk-based unlocks were also affected because they use the same Trip API.
- Trip completions remained available, and riders could return bicycles normally.
- 214 riders were incorrectly charged a temporary authorization hold after an unlock attempt failed. All holds were released within 24 hours.
- Customer support volume increased to approximately six times the normal evening rate.

No rider account data, payment credentials, or location history was exposed.

## Timeline

All times are in Eastern Daylight Time on July 8, 2026.

- **16:30** — Version `trip-api-2026.07.08.3` began rolling out to 10% of production instances.
- **16:36** — Automated deployment checks passed. Error rates remained below the configured alert threshold during the canary window.
- **16:40** — Deployment expanded to 100% of Trip API instances.
- **16:42** — Database query volume began increasing as the evening commute started.
- **16:47** — The first customer reports of failed unlocks reached support.
- **16:51** — Trip-start latency exceeded the service-level objective, but no page was triggered because the alert required a 15-minute sustained breach.
- **16:58** — Database connection utilization reached 100%. Trip-start failures increased rapidly.
- **17:03** — The on-call engineer was paged for elevated HTTP 503 responses.
- **17:09** — The incident was declared SEV-1, and an incident channel was opened.
- **17:17** — Engineers identified database connection exhaustion in the station authorization service.
- **17:25** — The database connection limit was temporarily increased by 20%. Error rates decreased briefly but rose again as traffic continued to grow.
- **17:34** — The team identified a sharp drop in authorization-cache hit rate following the Trip API deployment.
- **17:43** — A rollback of `trip-api-2026.07.08.3` began.
- **17:51** — The rollback completed, but several authorization-service instances remained unhealthy because their connection pools had not recovered.
- **18:04** — Authorization-service instances were restarted in batches.
- **18:18** — Cache hit rate returned to normal, and successful unlocks began recovering.
- **18:31** — Trip-start error rates fell below 5%.
- **18:45** — All station clusters reported healthy unlock activity.
- **18:59** — Error rates and latency returned to baseline. The incident was marked resolved.
- **20:20** — A reconciliation job identified 214 temporary payment holds associated with failed unlock attempts.
- **July 9, 11:30** — All affected payment holds had been released.

## Root Cause

The immediate cause was an incompatible change to the station authorization cache key.

Before the deployment, the Trip API requested authorization records using keys in this format:

`station:{station_id}:vehicle:{vehicle_id}`

The deployed version added the regional identifier to support an upcoming multi-region expansion:

`region:{region_id}:station:{station_id}:vehicle:{vehicle_id}`

The authorization service had not yet been updated to populate cache entries using the new format. As a result, nearly every request from the new Trip API version was treated as a cache miss.

Under normal conditions, approximately 94% of authorization requests are served from the cache. During the incident, the cache hit rate fell below 8%. Each cache miss generated multiple database queries to verify the bicycle, station, rider eligibility, and pricing plan. Evening commute traffic amplified this increase until the database connection pools were exhausted.

Once the pools were saturated, requests queued until they timed out. Retried requests from mobile applications and kiosks added further load, prolonging the failure.

## Contributing Factors

- The cache-key change was considered backward compatible during review because both formats contained the same identifiers. End-to-end compatibility with the authorization service was not tested.
- The canary deployment occurred before peak traffic and did not generate enough authorization requests to expose the reduced cache hit rate.
- Deployment checks monitored HTTP error rate and latency but did not evaluate cache hit rate or database connections per request.
- The latency alert required a 15-minute sustained breach, delaying the initial page.
- Mobile clients automatically retried failed unlock requests three times without exponential backoff or jitter.
- The authorization service did not limit concurrent database work when its cache was unhealthy.
- The rollout plan for the multi-region project did not document the required deployment order across dependent services.
- Application instances did not recover cleanly after database capacity was restored because timed-out connections remained in their pools.

## Resolution and Recovery

The team rolled back the Trip API to the previous version, restoring the original cache-key format. Engineers then restarted authorization-service instances in batches to clear unhealthy database connections and temporarily increased database capacity to accelerate recovery.

After trip-start traffic returned to normal, the team ran payment and trip reconciliation jobs. Temporary authorization holds created by failed requests were identified and released. No incomplete trips required manual closure.

## What Went Well

- Trip completion and bicycle return flows were isolated from the affected authorization path and remained available.
- The incident team identified the cache hit-rate regression within 40 minutes of the initial page.
- Rollback procedures were automated and completed without introducing additional failures.
- Payment reconciliation correctly detected every temporary hold associated with a failed unlock attempt.
- Operations staff provided status updates to customer support and regional transit partners throughout the incident.

## What Went Poorly

- Monitoring detected the symptoms but did not identify the cache regression during the canary deployment.
- The alerting delay allowed database saturation to become severe before the on-call engineer was paged.
- Customer-facing error messages incorrectly suggested that bicycles were unavailable.
- Retry behavior from clients significantly increased load during the outage.
- The team initially treated increased database capacity as a sufficient mitigation, delaying the rollback by approximately 18 minutes.
- Service recovery required manual instance restarts after the rollback.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---:|
| Add a contract test covering cache-key compatibility between the Trip API and authorization service | Trip Platform | P0 | July 22, 2026 |
| Restore multi-region support using a versioned cache-key schema with a dual-read migration period | Trip Platform | P0 | July 29, 2026 |
| Block deployments when authorization cache hit rate drops by more than 10 percentage points during canary analysis | Reliability Engineering | P0 | July 24, 2026 |
| Page when trip-start failures exceed 5% for five minutes | Reliability Engineering | P0 | July 18, 2026 |
| Add exponential backoff, jitter, and a retry budget to unlock requests in mobile and kiosk clients | Client Applications | P1 | August 12, 2026 |
| Add concurrency limits and load shedding for authorization database queries | Identity and Authorization | P1 | August 5, 2026 |
| Automatically recycle unhealthy database connections after repeated timeouts | Identity and Authorization | P1 | July 31, 2026 |
| Create a dependency-aware deployment checklist for changes spanning multiple services | Engineering Productivity | P1 | July 26, 2026 |
| Add synthetic trip-start tests at representative peak load during canary deployments | Quality Engineering | P1 | August 9, 2026 |
| Replace generic unlock failures with distinct messages for unavailable bikes, authorization failures, and service outages | Client Applications | P2 | August 20, 2026 |
| Run a game-day exercise simulating cache failure and database connection exhaustion | Reliability Engineering | P2 | September 4, 2026 |

## Lessons Learned

Cache schemas are service contracts and must be versioned, tested, and deployed with the same care as public APIs. Successful low-volume canary checks were insufficient because they validated availability without validating resource efficiency. Future deployment gates will include dependency-specific health indicators, including cache performance and database work per request, so regressions are detected before they reach peak traffic.