# Postmortem: Regional Bike-Sharing Unlock and Return Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 18 minutes  
**Severity:** SEV-1  
**Services affected:** Mobile unlocks, kiosk rentals, trip completion, station availability  
**Regions affected:** Greater Toronto and Hamilton Area  
**Status:** Resolved

## Summary

On July 24, 2026, a deployment to the station-state ingestion service caused duplicate dock events to be processed as new events. This produced sustained write amplification in the trip database, exhausted its connection pool, and prevented the rental API from reliably starting or ending trips.

During the incident, many riders could not unlock bikes. Some riders who returned bikes received an “active trip” message even though the dock had physically secured the bike. Station availability data was also delayed or incorrect.

We restored service by rolling back the ingestion deployment, temporarily reducing nonessential database traffic, and replaying validated dock events from the message queue. No trips or payment records were permanently lost.

## Customer Impact

From 08:07 to 10:25 EDT:

- 31,842 rental attempts were made.
- 12,406 rental attempts failed or timed out.
- 4,917 completed physical returns remained open in the customer application for more than five minutes.
- 1,126 riders were temporarily charged overage fees while their returns were pending.
- Station availability data was delayed by up to 23 minutes.
- Customer support received 3,284 contacts, approximately nine times the normal morning volume.
- 76 stations with older kiosk firmware displayed a generic “system unavailable” message.

All incorrect overage charges were automatically reversed by 14:40 EDT. Riders whose unlock attempts failed were not charged.

## Detection

Automated monitoring detected elevated rental API latency at 08:09 EDT. The on-call engineer was paged at 08:11 after the five-minute error-rate threshold was exceeded.

Our initial alert identified API saturation but did not identify the ingestion service as the source of the database load. This delayed diagnosis by approximately 20 minutes.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 07:45 | Version `station-ingest 4.18.0` was deployed to 10% of production consumers. Initial health checks passed. |
| 07:55 | Deployment reached 100%. Database write volume began increasing gradually. |
| 08:03 | A scheduled station firmware reconciliation caused approximately 1.8 million historical dock events to be republished. |
| 08:07 | Duplicate events began bypassing ingestion deduplication. Rental API latency exceeded two seconds. |
| 08:09 | Monitoring detected elevated latency and database connection utilization. |
| 08:11 | The primary on-call engineer was paged. |
| 08:14 | Rental API error rate exceeded 20%. Incident declared SEV-1. |
| 08:19 | Customer support reported riders unable to end trips at multiple stations. |
| 08:27 | Responders scaled rental API instances from 24 to 48. This did not improve availability because database connections remained exhausted. |
| 08:34 | Database team identified a sixfold increase in writes from the station-state ingestion service. |
| 08:41 | Deployment rollback began. |
| 08:49 | Rollback completed, but queued duplicate events continued to generate excessive writes. |
| 08:55 | Consumers were paused and the reconciliation publisher was disabled. |
| 09:03 | Database connection utilization began falling. New unlock success rate recovered to 92%. |
| 09:12 | A filtering rule was deployed to discard events whose station sequence number had already been committed. |
| 09:26 | Ingestion consumers resumed at reduced concurrency. |
| 09:44 | Unlock success rate returned to normal. Return processing remained delayed while the backlog drained. |
| 10:08 | All current return events had been processed. Reconciliation of affected trips began. |
| 10:25 | Station availability and trip completion metrics returned to normal. Incident marked resolved. |
| 14:40 | Incorrect overage charges were reversed and affected riders were notified. |

## Root Cause

The direct cause was a regression in version `4.18.0` of the station-state ingestion service.

The service normally deduplicates station events using a composite key containing the station identifier, dock identifier, event type, and station sequence number. The deployment changed the internal representation of the station sequence number from a string to an integer to support newer station firmware.

For events from older firmware, leading zeros were removed during conversion. The deduplication lookup continued using the original string value, while the persistence path stored the normalized integer value. As a result, repeated events such as sequence `0001842` were not recognized as duplicates of stored sequence `1842`.

The defect remained latent during the initial rollout because normal event duplication rates are low. At 08:03, a scheduled reconciliation job republished historical station events after incorrectly identifying several stations as behind. The ingestion service treated most of these events as new and wrote updated station and trip state for each event.

This write amplification exhausted the primary database’s application connection limit. The rental API could still accept requests, but it frequently could not acquire a connection before its timeout, causing unlock and return operations to fail or remain pending.

## Contributing Factors

- The reconciliation job had no limit on the number or age of events it could republish in one run.
- The ingestion canary received only live traffic and was not tested against a representative duplicate-event workload.
- Database alerts focused on aggregate connection utilization and did not immediately identify the responsible client.
- Scaling the rental API increased connection demand and briefly worsened contention.
- The physical dock network and customer-facing trip state are eventually consistent, so secured returns appeared active until their events were processed.
- Legacy kiosks did not distinguish temporary backend failures from station hardware failures.
- The deployment checklist did not require validation across all supported station firmware formats.

## Resolution and Recovery

Responders rolled back the ingestion deployment, paused consumers, and disabled the reconciliation publisher. Because duplicate events were already queued, rollback alone was insufficient.

The team added a temporary server-side filter based on the last committed sequence number for each station and resumed ingestion at reduced concurrency. Once current events were processed, a reconciliation job compared physical dock state, station events, and open trips to close affected rentals.

A billing correction process identified trips whose physical return preceded their recorded return by more than five minutes. It reversed resulting overage fees and queued customer notifications.

## What Went Well

- Monitoring detected customer-facing degradation within two minutes.
- The station event queue retained all events, allowing controlled replay without data loss.
- Physical dock locking continued to operate independently of the cloud API.
- Existing billing correction tooling reversed all identified incorrect charges on the same day.
- Engineering, operations, customer support, and field teams joined a shared incident channel within 15 minutes.

## What Did Not Go Well

- The deployment passed unit, integration, and canary checks despite breaking deduplication for legacy sequence formats.
- The first mitigation focused on scaling the API rather than reducing database demand.
- Ownership of the reconciliation publisher was unclear, delaying its shutdown.
- Support agents lacked a prepared message explaining that physically secured bikes were safe even if trips appeared active.
- Station-level dashboards showed stale availability without clearly indicating data freshness.

## Action Items

| Action | Owner | Priority | Target Date |
|---|---|---:|---|
| Replace mixed string/integer sequence handling with one canonical sequence type across ingestion, storage, and deduplication. | Station Platform | P0 | August 7, 2026 |
| Add replay tests covering duplicate, out-of-order, and leading-zero sequence numbers for every supported firmware family. | Station Platform | P0 | August 7, 2026 |
| Add a database uniqueness constraint for station event identity and make event writes idempotent. | Data Platform | P0 | August 14, 2026 |
| Add rate, age, and volume limits to reconciliation event publication. | Fleet Systems | P0 | August 10, 2026 |
| Require reconciliation jobs to stop automatically when duplicate or write-amplification thresholds are exceeded. | Fleet Systems | P1 | August 21, 2026 |
| Add per-client database connection and write-rate dashboards with actionable alerts. | Reliability Engineering | P1 | August 14, 2026 |
| Reserve database capacity for trip start and trip end operations. | Data Platform | P1 | August 28, 2026 |
| Prevent API autoscaling from exceeding its allocated database connection budget. | API Platform | P1 | August 21, 2026 |
| Add data-freshness indicators to rider and operations station maps. | Rider Experience | P2 | September 11, 2026 |
| Update kiosk error messages to distinguish backend outages from dock hardware failures. | Embedded Systems | P2 | September 25, 2026 |
| Create a customer-support incident playbook for delayed trip completion and secured returns. | Customer Operations | P1 | August 7, 2026 |
| Automate detection and reversal of fees caused by delayed return processing. | Payments | P1 | August 21, 2026 |
| Run a load exercise simulating a full-region station event replay. | Reliability Engineering | P1 | September 4, 2026 |

## Prevention

The primary prevention strategy is to make station event processing idempotent at multiple layers. Application-level deduplication will remain, but correctness will no longer depend on one in-memory key transformation. Database constraints will reject duplicate event identities, and reconciliation publishers will be bounded so they cannot create uncontrolled load.

We will also isolate critical trip operations from background station processing by assigning explicit connection budgets and prioritizing trip-start and trip-end writes. Future station ingestion releases will be tested using recorded event distributions from every supported firmware generation, including duplicates and out-of-order delivery.