# Postmortem: Regional Bike-Sharing Network Outage

**Incident date:** 2026-06-18  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Services affected:** Mobile app unlocks, kiosk rentals, station inventory updates, operator dispatch dashboard

## Summary

On June 18, 2026, the regional bike-sharing network experienced a service outage that prevented many riders from starting trips and caused station inventory data to become stale across the mobile app, kiosks, and operator tooling.

The incident began after a routine backend deployment introduced a database connection leak in the trip reservation service. As traffic increased during the evening commute, the service exhausted the shared PostgreSQL connection pool. This caused cascading failures in trip creation, bike unlock requests, station status updates, and internal dispatch workflows.

The issue was mitigated by rolling back the deployment, restarting affected service instances, and temporarily increasing database connection limits. Full service was restored at 18:42 local time.

## Customer Impact

- Approximately 14,800 riders opened the app during the incident window.
- 6,230 unlock attempts failed or timed out.
- 1,120 kiosk rental attempts failed.
- Station inventory data was delayed by up to 35 minutes.
- Some riders were shown bikes as available when they were already rented or unavailable.
- 312 active support tickets were created during and shortly after the incident.
- No completed trip billing records were lost, but 84 trips required manual fare adjustment due to delayed trip-start confirmation.

## Timeline

All times are local.

| Time | Event |
|---|---|
| 15:55 | Version `trip-reservation-service@2026.06.18.3` deployed to production. |
| 16:08 | First increase in database connection usage detected. No alert fired because usage remained below threshold. |
| 16:31 | Mobile unlock latency exceeded 3 seconds at the 95th percentile. |
| 16:44 | First customer reports of failed bike unlocks received by support. |
| 16:51 | Automated alert fired for elevated trip creation error rate. |
| 16:56 | On-call engineer began investigation. |
| 17:07 | Database connection pool saturation identified. |
| 17:14 | Trip reservation service restarted, temporarily reducing error rate. |
| 17:26 | Error rate increased again as connection pool saturation returned. |
| 17:34 | Incident declared SEV-2. Engineering, SRE, support, and operations joined incident bridge. |
| 17:42 | Recent deployment identified as likely trigger. |
| 17:51 | Rollback initiated for trip reservation service. |
| 18:06 | Rollback completed across all production regions. |
| 18:13 | Database connection utilization began returning to normal. |
| 18:21 | Station inventory update backlog began draining. |
| 18:42 | Unlock success rate returned to normal operating range. Incident resolved. |
| 19:10 | Customer support began fare adjustment review for affected trips. |

## Root Cause

The outage was caused by a database connection leak introduced in the trip reservation service.

A new code path added during the deployment opened a transaction to validate bike availability and reserve the bike for a rider. When validation failed due to a race condition, such as two riders attempting to reserve the same bike at nearly the same time, the function returned early without closing the transaction.

During normal low-traffic periods, the leak accumulated slowly. During the evening commute, failed reservation attempts increased significantly, causing service instances to consume all available PostgreSQL connections. Because the affected database pool was shared with station inventory writes and kiosk rental flows, those systems also degraded.

## Contributing Factors

- The connection pool alert threshold was too high and only alerted after user-facing errors had already started.
- The new transaction path did not have a test covering early returns on validation failure.
- The trip reservation service and station inventory service shared the same database connection pool.
- The deployment occurred shortly before peak commute traffic.
- Kiosk and mobile unlock failures surfaced as generic timeout errors, slowing diagnosis.

## Detection

The incident was initially detected through automated alerting on elevated trip creation error rates. Customer support reports arrived several minutes earlier, but they were not automatically correlated with backend service health signals.

## Resolution

The team restored service by:

1. Restarting affected trip reservation service instances to release leaked connections.
2. Rolling back the most recent trip reservation deployment.
3. Temporarily increasing PostgreSQL connection limits.
4. Monitoring database pool recovery and station inventory backlog drain.
5. Manually reviewing trips with delayed start confirmation for billing corrections.

## What Went Well

- Rollback completed cleanly without data migration concerns.
- Billing records remained consistent.
- Operations staff were able to rebalance high-demand stations once inventory data recovered.
- Support quickly identified common rider symptoms and routed reports to the incident channel.

## What Went Wrong

- The issue reached customers before alerting triggered.
- The deployment window overlapped with peak commute demand.
- Shared database capacity allowed one service failure to affect multiple rider-facing workflows.
- Error messages in the app and kiosk did not distinguish service timeouts from unavailable bikes.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add automated tests for transaction cleanup on all early-return paths. | Backend | P0 | 2026-06-21 |
| Add transaction timeout and leak detection metrics to trip reservation service. | Backend | P0 | 2026-06-24 |
| Lower database connection pool alert thresholds and add rate-of-change alerts. | SRE | P0 | 2026-06-24 |
| Separate connection pools for trip reservations, station inventory, and kiosk rentals. | Platform | P1 | 2026-07-05 |
| Block non-emergency deployments within 90 minutes of weekday commute peaks. | Engineering Mgmt | P1 | 2026-06-28 |
| Improve mobile and kiosk error messages for unlock failures. | Rider Apps | P2 | 2026-07-12 |
| Add support ticket spike correlation to incident alerting. | Support Ops | P2 | 2026-07-19 |

## Prevention

The primary prevention work is to make transaction lifecycle errors easier to detect before production and less capable of affecting unrelated services when they do occur. The most important changes are stricter transaction cleanup tests, earlier connection pool alerting, and isolation of database capacity between critical rider workflows.