# Postmortem: Station Unlock and Trip-Start Outage

**Incident date:** July 18, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Affected region:** Greater Lakeshore service area  
**Status:** Resolved

## Summary

On July 18, riders across the Greater Lakeshore bike-sharing network were unable to reliably unlock bicycles or start trips. The incident began at 07:42 EDT after a routine deployment caused the station gateway service to exhaust its database connection pool.

The resulting backlog delayed or rejected unlock requests from the mobile application, kiosk terminals, and member key readers. Existing trips and bicycle returns continued to function, although some completed trips appeared active in the application until delayed events were processed.

Service was restored at 10:29 EDT after the deployment was rolled back, the gateway connection pool was reconfigured, and queued station events were drained.

## Customer Impact

During the incident:

- 18,642 unlock attempts failed or timed out.
- 11,306 unique riders were affected.
- 63% of trip-start requests failed during the peak of the outage.
- 2,184 completed trips remained temporarily visible as active.
- 417 riders were incorrectly shown potential overtime charges; these charges were not collected.
- Customer support received 1,276 contacts, approximately nine times the normal volume.
- Public transit transfer stations experienced the highest impact during the morning commute.

Bicycle returns remained available because docks store return events locally and release the locking mechanism without waiting for the central platform. No bicycles were left physically unlocked as a result of the incident.

## Timeline

All times are in Eastern Daylight Time.

- **07:31** — Version `station-gateway 4.18.0` deployed to 10% of production instances.
- **07:37** — Automated health checks passed, and deployment expanded to all instances.
- **07:42** — Database connection utilization began rising rapidly.
- **07:45** — First elevated-latency alert fired for the trip-start API.
- **07:49** — On-call engineer acknowledged the alert and began investigating.
- **07:53** — Unlock success rate fell below 70%.
- **07:57** — Customer support reported a sharp increase in failed-unlock complaints.
- **08:02** — Incident declared SEV-1; engineering, operations, support, and communications leads joined the response.
- **08:09** — Responders identified database connection exhaustion in the station gateway service.
- **08:17** — The team reduced gateway traffic by disabling nonessential station-status refreshes.
- **08:26** — Unlock success briefly improved to 58% but declined as queued requests retried.
- **08:34** — Deployment `4.18.0` identified as the likely trigger.
- **08:41** — Rollback initiated.
- **08:52** — Rollback completed, but recovery remained slow because stale connections and retry traffic continued consuming capacity.
- **09:03** — Gateway instances restarted in controlled batches to clear stale connections.
- **09:21** — Unlock success rate recovered above 85%.
- **09:38** — Database connection utilization returned to normal.
- **09:47** — Unlock success rate exceeded 98%; queued return events continued processing.
- **10:12** — Delayed trip records and account states were reconciled.
- **10:29** — Incident resolved after metrics remained healthy for 30 minutes.
- **12:10** — Incorrect overtime warnings removed from affected accounts.
- **16:30** — Rider credits issued and customer notification completed.

## Root Cause

Version `4.18.0` introduced a new eligibility check intended to prevent bicycles marked for maintenance from being assigned to riders. The implementation performed an additional database query for every candidate bicycle returned during an unlock request.

The query used a helper that opened its own database connection instead of reusing the transaction-scoped connection already held by the request. Under normal traffic, the additional connections were released quickly enough to avoid visible impact. During the morning commute, however, each unlock request evaluated multiple bicycles and temporarily held several connections at once.

This increased connection demand beyond the database pool limit. Requests waiting for connections exceeded their API deadlines, causing clients and station gateways to retry. Those retries further increased demand, creating a feedback loop that exhausted the pool across all gateway instances.

## Contributing Factors

- Load testing modeled average daily traffic but did not include the morning commuter burst or client retry behavior.
- The canary deployment was evaluated for six minutes, which was too short to capture connection accumulation under sustained load.
- Deployment health checks measured process availability and error rate but did not include database pool saturation.
- The retry policy used fixed intervals without jitter and allowed retries after the server had already queued the original request.
- Station-status refreshes and unlock requests shared the same database pool, allowing lower-priority traffic to consume capacity needed for trip starts.
- The new query path lacked a production-like performance test and per-request connection-count metric.
- The rollback restored application code but did not immediately terminate connections opened by affected instances, delaying recovery.

## Resolution and Recovery

Responders rolled back `station-gateway 4.18.0` and restarted gateway instances in batches to release stale connections without causing a complete service interruption. Nonessential station-status refreshes were temporarily disabled to reserve database capacity for trip starts and returns.

After the gateway stabilized, operations processed the station-event backlog and ran account reconciliation to close trips whose return events had been delayed. Billing was paused for affected trips until reconciliation completed. No incorrect overtime charges were collected.

## What Went Well

- Alerts detected elevated latency within three minutes of customer-visible impact.
- Station hardware continued accepting returns in offline mode.
- Billing safeguards prevented temporary trip-state errors from becoming charges.
- Engineering and operations established a shared incident channel quickly.
- Account reconciliation completed the same day without manual changes to individual trip records.

## What Went Poorly

- The initial canary window was too short for this failure mode.
- Dashboards showed database utilization but not connection demand by endpoint.
- Client retries amplified the incident and made recovery less predictable.
- The rollback runbook did not document how to clear stale gateway connections.
- Customer messaging began 46 minutes after the incident started.
- Support agents initially lacked guidance for riders seeing active trips after returning bicycles.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Change the maintenance eligibility check to reuse the request-scoped database connection | Station Platform | P0 | July 22, 2026 |
| Add tests that fail when an unlock request acquires more than one database connection | Station Platform | P0 | July 25, 2026 |
| Implement exponential backoff with jitter and retry budgets for unlock requests | Mobile and Firmware | P0 | August 8, 2026 |
| Separate trip-start traffic from station-status refreshes using independent connection pools | Infrastructure | P1 | August 15, 2026 |
| Alert on connection-pool wait time, saturation, and connections acquired per endpoint | Observability | P1 | August 5, 2026 |
| Extend canary observation to cover at least one representative traffic peak | Release Engineering | P1 | August 1, 2026 |
| Add commuter-burst and retry-storm scenarios to preproduction load tests | Performance Engineering | P1 | August 12, 2026 |
| Add automated load shedding for nonessential station-status requests | Station Platform | P1 | August 19, 2026 |
| Update the rollback runbook with stale-connection cleanup and recovery validation | Site Reliability Engineering | P1 | August 4, 2026 |
| Publish customer communications within 15 minutes of a confirmed SEV-1 rider impact | Customer Operations | P1 | August 2, 2026 |
| Add a support playbook for delayed returns, active-trip displays, and billing holds | Customer Support | P2 | August 2, 2026 |
| Conduct a quarterly offline-return and reconciliation exercise | Operations | P2 | September 30, 2026 |

## Lessons Learned

A service can pass basic health checks while exhausting a shared dependency under sustained load. For rider-critical operations, deployment validation must account for peak demand, retries, and resource use per request—not only application error rates.

We also need to treat recovery behavior as part of system design. Retries, stale connections, and queued events extended this incident after the faulty release had been removed. Future changes will include explicit resource limits, traffic prioritization, and recovery testing as release requirements.