# Postmortem: Regional Bike-Share Unlock Outage

**Incident date:** 2026-07-09  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Final  
**Services affected:** Mobile app unlocks, kiosk rentals, dock command dispatch, station availability updates

## Summary

On July 9, the regional bike-sharing network experienced a widespread outage that prevented many riders from unlocking bikes through the mobile app and station kiosks. The issue affected 61% of stations across the metro region during the morning commute.

The outage was caused by a faulty configuration rollout that directed dock command traffic to a Redis replica pool that was not authorized to perform writes. Because dock unlock commands require a short-lived reservation token to be written before the command is sent, unlock requests failed even though the mobile app, payment system, and station cellular connectivity remained mostly healthy.

## Customer Impact

- Approximately 18,400 unlock attempts failed.
- 7,900 riders were directly affected.
- 312 active rentals could not be ended automatically until station connectivity recovered.
- Customer support volume increased by 4.6x during the incident window.
- Some riders were charged incorrectly for extended trips; these charges were reversed after reconciliation.
- Station availability data was stale for up to 35 minutes in parts of the region.

## Timeline

All times are in ET.

| Time | Event |
| --- | --- |
| 07:28 | Configuration rollout began for `dock-command-api` to support regional Redis pool separation. |
| 07:36 | First increase in unlock failures detected by application metrics. |
| 07:41 | On-call engineer paged for elevated `unlock_command_failed` rate. |
| 07:46 | Customer support reported multiple complaints from downtown stations. |
| 07:52 | Incident declared SEV-1. |
| 08:03 | Engineers confirmed payment authorization and rider account services were healthy. |
| 08:11 | Dock command logs showed Redis `READONLY` errors during reservation-token writes. |
| 08:18 | Rollout paused, but affected pods continued using cached configuration. |
| 08:31 | `dock-command-api` pods restarted in one availability zone as a test. Unlock success rate improved locally. |
| 08:44 | Full rollback approved and initiated. |
| 09:02 | All `dock-command-api` pods restarted with prior Redis primary configuration. |
| 09:21 | Unlock success rate returned to baseline. |
| 09:45 | Station availability backlog drained. |
| 10:03 | Incident resolved. |
| 12:30 | Billing reconciliation job completed and incorrect extended-trip charges were reversed. |

## Root Cause

The immediate cause was an invalid production configuration value for `dock-command-api`.

The service was changed to read Redis connection details from a new regional configuration map. In the production rollout, the `primary_endpoint` and `replica_endpoint` values were reversed for the central region. As a result, `dock-command-api` attempted to write reservation tokens to a Redis replica.

Reservation-token writes are required before a dock receives an unlock command. When Redis returned `READONLY` errors, the service treated the failures as command preparation failures and did not send unlock commands to stations.

The issue was not caught before deployment because staging uses a single-node Redis instance and does not enforce primary/replica write behavior.

## Contributing Factors

- The configuration change was deployed during the morning commute window.
- The service cached Redis configuration at process startup, so pausing the rollout did not fix already-running pods.
- Alerts detected unlock failures, but there was no specific alert for Redis `READONLY` errors.
- Staging did not match production Redis topology.
- The deployment checklist did not require validation of regional endpoint roles.

## What Went Well

- The on-call engineer was paged within five minutes of customer-visible impact.
- Metrics clearly showed that failures were isolated to unlock command preparation.
- The rollback was straightforward once the bad configuration was identified.
- Billing reconciliation successfully corrected affected trips the same day.

## What Went Poorly

- The bad configuration passed review because endpoint names were similar and manually entered.
- The first mitigation, pausing rollout, did not help because running pods retained cached values.
- Customer-facing status updates lagged behind the incident by approximately 25 minutes.
- Support agents initially lacked a clear script for riders with bikes that would not dock or unlock.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add automated validation that Redis primary endpoints accept writes before production rollout. | Platform Engineering | 2026-07-18 | Open |
| Update staging Redis to use primary/replica topology. | Infrastructure | 2026-07-25 | Open |
| Add alert for Redis `READONLY` errors in `dock-command-api`. | Observability | 2026-07-17 | Open |
| Move commute-hour deployment freeze into CI/CD policy. | Release Engineering | 2026-07-19 | Open |
| Change `dock-command-api` to reload Redis configuration without process restart. | Mobility Backend | 2026-08-02 | Open |
| Add deployment checklist item for regional endpoint verification. | SRE | 2026-07-16 | Open |
| Improve customer status-page automation for SEV-1 incidents. | Customer Operations | 2026-07-24 | Open |
| Create support macros for failed unlocks, failed returns, and billing reversals. | Customer Support | 2026-07-17 | Open |

## Prevention

Future regional configuration changes will require automated endpoint validation, production-like staging coverage, and deployment outside peak commute periods. The dock command path will also be updated to fail faster with clearer diagnostics when its reservation store is unavailable or misconfigured.