# Postmortem: Regional Bike-Share Unlock and Docking Outage

**Date:** 2026-06-18  
**Duration:** 2 hours 14 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On June 18, 2026, the regional bike-sharing network experienced a service outage that prevented riders from unlocking bikes and intermittently delayed dock status updates across the Metro North region. Existing rides could continue, but many riders were unable to start new trips or reliably end trips at smart docks.

The outage was caused by a failed database migration that introduced lock contention on the trip sessions table during peak commute hours. The issue overloaded the booking API and caused cascading timeouts in the mobile app, kiosk, and dock controller services.

## Customer Impact

- Approximately 18,400 riders were affected.
- 7,900 unlock attempts failed.
- 2,300 trip-end events were delayed by more than 5 minutes.
- 410 riders were incorrectly shown as still on active trips after docking.
- Customer support volume increased by 6.8x during the incident.
- No riders were overcharged; delayed trips were corrected during reconciliation.

## Timeline

All times in Eastern Time.

| Time | Event |
|---|---|
| 07:42 | Database migration begins as part of scheduled release `2026.06.18.1`. |
| 07:51 | Elevated latency detected on booking API `POST /trips/start`. |
| 07:56 | Error rate exceeds alert threshold; on-call engineer paged. |
| 08:03 | Customer support reports widespread unlock failures in Metro North. |
| 08:09 | Engineering declares SEV-1 incident. |
| 08:14 | Mobile app and kiosk traffic begins retrying aggressively, increasing API load. |
| 08:21 | Migration identified as holding locks on `trip_sessions` and `dock_events`. |
| 08:29 | Release rollback started. |
| 08:41 | Rollback completes, but database lock queue remains saturated. |
| 08:52 | API retry limits reduced through feature flag. |
| 09:08 | Read replicas recover; dock status updates begin catching up. |
| 09:31 | Manual termination of blocked migration process completed. |
| 09:44 | Unlock success rate returns above 99%. |
| 09:56 | Delayed trip-end reconciliation job completes. |
| 10:06 | Incident resolved. |

## Root Cause

A database migration added a new nullable column and backfilled historical rows in the `trip_sessions` table. The migration was expected to run online, but the backfill query performed large batch updates without an effective index on `region_id` and `ended_at`.

During peak commute traffic, the migration caused prolonged row-level locking and increased write latency. The booking API depends on low-latency writes to create and update trip sessions. As write latency increased, API requests timed out. Client retries from the mobile app, kiosks, and dock controllers amplified the load, which further delayed recovery.

## Contributing Factors

- The migration was scheduled during a high-traffic commute window.
- The migration plan was reviewed for schema safety but not for production data volume.
- Client retry behavior did not include enough jitter or backoff.
- The booking API shared the same database write pool as dock event ingestion.
- Alerting detected API failure rate quickly but did not identify database lock contention directly.

## What Went Well

- On-call response began within 5 minutes of the first page.
- Feature flags allowed retry limits to be reduced without a new deploy.
- Trip reconciliation corrected delayed and incorrect active-trip states.
- No billing data was lost.

## What Went Wrong

- The release process allowed a high-risk data migration during peak usage.
- Rollback did not stop the already-running migration.
- Dock event ingestion and trip creation were not isolated from each other.
- Customer-facing status updates lagged behind the incident declaration by 18 minutes.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---|---|
| Block non-emergency migrations during weekday commute windows | Platform | 2026-06-25 | Open |
| Add migration review checklist for lock risk and data volume | Database Team | 2026-06-28 | Open |
| Require dry-run query plans for production backfills | Database Team | 2026-07-02 | Open |
| Add lock contention alerts for critical tables | SRE | 2026-07-05 | Open |
| Separate dock event ingestion write pool from trip booking writes | Backend | 2026-07-12 | Open |
| Add exponential backoff and jitter to app, kiosk, and dock retries | Client Platform | 2026-07-15 | Open |
| Improve incident status page automation | Support Engineering | 2026-07-09 | Open |
| Run reconciliation job automatically after trip-state incidents | Backend | 2026-07-16 | Open |

## Prevention

Future migrations that touch high-volume trip or dock tables must use small batches, explicit throttling, production-sized dry runs, and an approved execution window. Critical customer workflows will also be isolated so that dock ingestion delays cannot exhaust booking capacity during database degradation.