# Postmortem: Trip Start Failures During Morning Commute **Incident date:** July 17, 2026 **Duration:** 1 hour 48 minutes **Severity:** SEV-1 **Services affected:** Mobile trip start, kiosk rentals, customer support tools **Region:** Greater Lakeshore bike-sharing network **Status:** Resolved ## Summary From 07:42 to 09:30 EDT, riders across the Greater Lakeshore network experienced elevated failures when attempting to unlock bicycles. The incident affected both mobile-app and station-kiosk rentals during the morning commute. A deployment to the station inventory service introduced a database query that acquired row locks in an inconsistent order. Morning demand caused concurrent trip-start requests and bicycle-status updates to deadlock. Automatic retries amplified database load, exhausted the API connection pool, and caused otherwise healthy requests to time out. We restored service by disabling the new inventory reconciliation path, reducing retry concurrency, and restarting saturated API workers. No active trips were terminated, and bicycles already in use could be returned normally. ## Customer Impact During the incident: - 18,742 trip-start attempts were made. - 7,916 attempts failed or timed out. - Approximately 5,300 unique riders were affected. - Mobile-app unlock success fell from 99.4% to a low of 41.8%. - Kiosk rental success fell from 98.9% to a low of 53.2%. - Trip-end processing remained available, although 612 completed trips appeared active in the app for up to 27 minutes because receipt generation was delayed. - 1,486 riders contacted customer support, increasing median response time from 4 minutes to 38 minutes. - 327 riders were incorrectly shown a temporary authorization hold after retrying a failed rental. All holds were released automatically within 24 hours; no duplicate charges were captured. The cities of Northport, East Junction, and Lakeshore Centre saw the highest failure rates because those zones had the greatest concentration of simultaneous commuter demand. ## Detection The incident was first detected by an alert for elevated trip-start latency at 07:46 EDT, four minutes after customer impact began. A separate unlock-success alert fired at 07:49. The initial alert identified API latency but did not distinguish between station connectivity failures and backend transaction failures. This delayed identification of the inventory database as the primary bottleneck. ## Timeline All times are in EDT. - **07:28** — Version `inventory-service 2026.07.17.1` begins deployment to production. - **07:36** — Deployment completes. Initial health checks and low-volume synthetic rentals succeed. - **07:42** — Morning demand increases. Database deadlocks begin in the new inventory reconciliation query. - **07:46** — Trip-start latency alert fires. On-call engineer begins investigation. - **07:49** — Unlock-success rate drops below 80%; SEV-1 incident is declared. - **07:54** — Operations reports that station controllers are online, ruling out a regional cellular-network failure. - **08:01** — Engineers identify API connection-pool saturation but initially attribute it to increased support-tool traffic. - **08:08** — Customer communications are posted in the mobile app and on the service-status page. - **08:14** — Database metrics reveal a sharp increase in deadlocks and transaction retries from the inventory service. - **08:21** — The team pauses background fleet-balancing jobs to reduce database load. Unlock success briefly improves to 68%. - **08:29** — Engineers correlate the deadlocks with the newly deployed reconciliation path. - **08:36** — Rollback begins. The first rollback attempt stalls because several instances are waiting on database connections. - **08:45** — Retry concurrency is reduced through a runtime configuration change. - **08:52** — API workers begin recovering as database connection pressure falls. - **09:01** — The reconciliation feature is disabled across all instances. - **09:08** — Mobile and kiosk unlock success returns above 95%. - **09:18** — Delayed trip receipts and authorization reconciliation begin processing normally. - **09:30** — Success rate and latency remain within service-level objectives for 20 minutes; the incident is marked resolved. - **10:12** — Customer support receives a list of affected accounts and approved remediation guidance. - **13:40** — All delayed trip records are reconciled. - **July 18, 11:15** — Verification confirms that no duplicate rental charges were captured. ## Root Cause The deployment introduced a reconciliation operation intended to prevent a bicycle from being assigned when its station-reported state differed from the central inventory record. The new operation updated the bicycle record and station dock record within one transaction. The trip-start path locked the bicycle row first and the dock row second. A pre-existing station synchronization path locked the same records in the opposite order. Under normal load, the overlap was uncommon. During the morning commute, concurrent requests frequently selected the same station records, producing database deadlocks. The database correctly aborted one transaction in each deadlock. However, the inventory service automatically retried aborted transactions up to five times without randomized backoff. These retries increased query volume and lock contention. As transactions waited, service instances consumed all available database connections. Trip-start requests then timed out before an unlock command could be issued. The immediate technical root cause was inconsistent database lock ordering between the new reconciliation path and the existing station synchronization path. The outage became regional because aggressive retries and an unbounded request queue amplified the contention. ## Contributing Factors - Load testing modeled average regional traffic but not concentrated demand at high-volume stations. - The staging database contained fewer bicycles per station and did not reproduce production lock contention. - Deadlock metrics existed at the database level but were not included in the inventory-service dashboard. - Retry behavior used fixed, immediate retries instead of exponential backoff with jitter. - The deployment proceeded to all production instances within eight minutes, leaving insufficient observation time at partial traffic. - The rollback procedure depended on healthy database connections, which were already exhausted. - Synthetic rental checks ran serially and did not exercise concurrent trip starts and station synchronization. - Mobile clients allowed repeated unlock attempts while the first request was still pending, adding traffic during the incident. ## Resolution and Recovery The team disabled the new reconciliation path using a feature flag, reduced transaction retry concurrency, and restarted API workers with saturated connection pools. Background fleet-balancing jobs were paused until the primary database recovered. After unlock success returned to normal, the team replayed delayed trip events and reconciled payment authorizations. Customer support proactively contacted riders with duplicate pending authorizations or trips that appeared active after return. ## What Went Well - Station-controller telemetry remained available and helped rule out a field-network outage quickly. - The reconciliation feature could be disabled without deploying new code. - Trip-end events were durably queued, preventing loss of completed-trip data. - Payment capture required a confirmed unlock event, so failed rentals did not produce duplicate charges. - Engineering, operations, payments, and customer support joined the incident channel within 20 minutes of declaration. ## What Did Not Go Well - Service-level alerts did not identify database deadlocks as the likely source. - The first rollback attempt was delayed by the same connection exhaustion causing the outage. - The status-page update occurred 26 minutes after customer impact began. - Support agents initially lacked guidance about pending authorization holds. - The production rollout was too fast for a change affecting a high-volume transactional path. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---| | Enforce a consistent bicycle-then-dock lock order in all inventory transactions | Inventory Platform | P0 | July 31, 2026 | | Add exponential backoff, jitter, and a retry budget for deadlocked transactions | Inventory Platform | P0 | July 31, 2026 | | Add per-service database connection limits and bounded request queues | Core Services | P0 | August 7, 2026 | | Alert on inventory-service deadlock rate and connection-pool saturation | Reliability Engineering | P0 | August 3, 2026 | | Add concurrent trip-start and station-sync scenarios to pre-production load tests | Quality Engineering | P1 | August 14, 2026 | | Expand staging data to match the size and station-density distribution of production | Data Platform | P1 | August 21, 2026 | | Change high-risk deployments to a 5% canary with a 30-minute observation window | Release Engineering | P1 | August 7, 2026 | | Make rollback and feature-flag controls independent of the primary application database | Platform Engineering | P1 | August 28, 2026 | | Prevent mobile clients from issuing duplicate trip-start requests while one is pending | Rider Applications | P1 | August 14, 2026 | | Add an automated status-page trigger for sustained regional unlock failures | Operations | P2 | August 21, 2026 | | Create a support playbook for delayed trips and payment authorization holds | Customer Operations | P2 | August 5, 2026 | | Run a failure-mode review for all workflows that update bicycle and dock records together | Architecture Group | P2 | August 28, 2026 | ## Lessons Learned Feature flags limited recovery time, but they did not compensate for insufficient concurrency testing and an overly rapid rollout. Changes to transaction boundaries and lock acquisition must be treated as high risk even when functional behavior appears unchanged. Future reviews for inventory workflows will explicitly evaluate lock ordering, retry amplification, queue bounds, and degraded behavior under station-level demand spikes.