# Postmortem: Regional Bike-Sharing Network Outage

## Summary

On 2026-06-18, the regional bike-sharing network experienced a service outage that prevented customers from unlocking bikes and ending trips across most stations in the North Valley region. The outage lasted 1 hour and 47 minutes, from 07:42 to 09:29 local time, during the morning commute.

The incident was caused by a database connection pool exhaustion in the trip orchestration service after a configuration change reduced the effective maximum number of available database connections. The service continued accepting requests but could not reliably read or write trip state, causing unlock requests, trip starts, and trip endings to fail or time out.

## Impact

- Approximately 18,400 customers were affected.
- 7,900 bike unlock attempts failed.
- 2,300 active trips could not be ended through the mobile app or station kiosk.
- 410 customers were incorrectly shown as still riding after returning bikes.
- Customer support received 1,260 contacts during the incident window.
- No bikes were permanently lost, but 37 bikes required manual reconciliation.
- Customers with affected trips were automatically refunded.

## Timeline

| Time | Event |
|---|---|
| 07:42 | Error rate for unlock requests increased above alert threshold. |
| 07:45 | On-call engineer acknowledged the paging alert. |
| 07:49 | Customer support reported a spike in calls from riders unable to unlock bikes. |
| 07:53 | Engineering confirmed elevated latency and timeout errors in the trip orchestration service. |
| 08:01 | Initial investigation focused on mobile API gateway health due to high request retries. |
| 08:14 | Database metrics showed connection pool saturation from trip orchestration workers. |
| 08:21 | Recent configuration deployment identified as the likely trigger. |
| 08:28 | Deployment rollback started. |
| 08:39 | Rollback completed, but stuck worker processes continued holding connections. |
| 08:47 | Trip orchestration workers were restarted region by region. |
| 09:03 | Unlock success rate recovered to normal levels. |
| 09:18 | Trip-ending backlog processing completed. |
| 09:29 | Incident declared resolved. |
| 10:15 | Customer refunds and trip corrections began. |
| 13:40 | All known billing and trip-state discrepancies reconciled. |

## Root Cause

A configuration change deployed at 07:35 reduced the trip orchestration service database connection pool from 80 connections per worker group to 20. The change was intended to apply only to a new batch reconciliation worker but was placed in a shared configuration block used by all trip orchestration workers.

During the morning commute, request volume increased as expected. With the reduced connection pool, trip orchestration workers exhausted available database connections and began queueing requests internally. Retries from mobile clients and station kiosks increased pressure on the service, causing unlock and trip-ending requests to time out.

The rollback restored the previous configuration, but several worker processes remained in a degraded state because they retained stale connection pool settings until restarted.

## Contributing Factors

- The configuration file grouped batch and real-time worker settings together without clear ownership boundaries.
- The deployment validation checked service startup but did not run a realistic unlock or trip-ending transaction.
- Alerts fired on API error rate but did not immediately identify database pool saturation.
- Client retry behavior amplified load during partial failures.
- The rollback procedure did not include restarting long-lived workers that cache configuration at startup.

## Detection

The incident was detected by automated monitoring when unlock request failures exceeded 5% for five consecutive minutes. Customer support reports confirmed the issue shortly afterward.

Detection was effective, but diagnosis took longer than expected because the first visible symptom was API timeout volume rather than database pool saturation.

## Resolution

Engineering rolled back the configuration change and restarted trip orchestration workers across the affected region. Once workers restarted with the correct pool size, unlocks and trip endings returned to normal. A reconciliation job corrected affected trip records, closed trips that had already physically ended, and issued refunds where appropriate.

## What Went Well

- Automated alerts detected the outage within minutes.
- The recent deployment was identified quickly once database metrics were reviewed.
- Regional worker restarts restored service without requiring a full platform restart.
- Trip reconciliation tools were able to correct customer accounts and billing records.

## What Went Poorly

- The configuration change affected a broader set of services than intended.
- Rollback did not fully restore service until workers were restarted manually.
- Customers with active trips received confusing app states during the incident.
- Support teams did not receive an incident summary until 42 minutes after detection.

## Action Items

| Action Item | Owner | Due Date | Status |
|---|---|---|---|
| Split batch worker and real-time trip orchestration configuration into separate files. | Platform Engineering | 2026-06-28 | Open |
| Add pre-deployment smoke tests for bike unlock, trip start, and trip end flows. | QA Engineering | 2026-07-05 | Open |
| Update rollback runbook to include restart steps for workers that cache configuration. | SRE | 2026-06-25 | Open |
| Add alerts for database connection pool saturation by service and region. | Observability | 2026-06-30 | Open |
| Reduce mobile and kiosk retry amplification with capped exponential backoff and jitter. | Client Engineering | 2026-07-12 | Open |
| Add customer-facing incident messaging in the app for unlock and return failures. | Product Engineering | 2026-07-19 | Open |
| Create an automated trip-state reconciliation job that runs during service recovery. | Payments Engineering | 2026-07-10 | Open |
| Improve support escalation workflow for regional service incidents. | Customer Operations | 2026-06-27 | Open |

## Prevention

To prevent recurrence, configuration ownership will be separated between real-time and batch systems, and deployments that affect trip orchestration will require transaction-level smoke tests before regional rollout. Rollback procedures will be updated to account for long-lived workers, and monitoring will be expanded to alert directly on resource saturation before customer-facing failures become widespread.