# Postmortem: Checkout Failures During Saturday Market Rush **Incident date:** July 25, 2026 **Duration:** 8:42–10:17 a.m. ET **Severity:** SEV-1 **Services affected:** Web and mobile checkout, vendor inventory **Status:** Resolved ## Summary During the Saturday morning market rush, customers were unable to complete purchases from participating farmers-market vendors. A database migration introduced row-level inventory locking that caused checkout requests to queue behind long-running cart reservations. As traffic increased, the checkout service exhausted its database connection pool and began returning errors. The incident lasted 95 minutes. Browsing remained available, but many carts displayed stale inventory, and checkout success fell as low as 18%. We restored service by disabling the new reservation path, terminating blocked database sessions, and increasing checkout capacity while the backlog cleared. ## Customer Impact Between 8:42 and 10:17 a.m. ET: - 4,812 checkout attempts failed or timed out. - 2,064 customers were affected. - 731 orders were completed successfully. - 186 customers were charged after seeing an error page; all corresponding orders were created, and no duplicate charges were identified. - 427 carts contained items that became unavailable before customers could retry. - 38 vendors temporarily oversold at least one product because inventory updates were delayed. - Estimated gross merchandise value lost was $96,000. Customers could still browse vendors and add items to carts. Vendor dashboards remained accessible, but inventory counts were delayed by up to 22 minutes. ## Timeline All times are in Eastern Time. - **8:30 a.m.** Most participating markets open. Traffic begins rising as expected. - **8:37 a.m.** The new inventory-reservation code path reaches 100% of checkout traffic following a staged rollout started the previous evening. - **8:42 a.m.** Checkout latency exceeds the five-second alert threshold. - **8:44 a.m.** The checkout error rate alert fires. The on-call engineer begins investigating. - **8:48 a.m.** Database connection usage reaches 100%. Checkout pods begin failing readiness checks and restarting. - **8:52 a.m.** Support reports multiple customer complaints about spinning payment screens and error pages. - **8:56 a.m.** The incident is declared SEV-1. Engineering, support, payments, and marketplace operations join the incident channel. - **9:03 a.m.** The team identifies a large number of database sessions waiting on inventory-row locks. - **9:09 a.m.** Automatic scaling increases checkout pods from 24 to 60. This does not improve throughput because the database connection pool remains exhausted. - **9:16 a.m.** The team pauses nonessential inventory reconciliation jobs to reduce database load. - **9:24 a.m.** Engineers trace the blocking queries to cart reservations created by the previous evening’s deployment. - **9:31 a.m.** A rollback is attempted but fails because the database migration changed a constraint required by the previous application version. - **9:39 a.m.** The team disables inventory reservations through a feature flag and returns checkout to the previous optimistic-inventory flow. - **9:47 a.m.** Blocked database sessions are terminated in batches. Checkout success begins recovering. - **9:55 a.m.** Payment reconciliation confirms that orders exist for customers charged during failed requests. - **10:06 a.m.** Checkout success exceeds 95%; latency remains elevated while queued requests drain. - **10:17 a.m.** Error rates and latency return to normal. The incident is marked resolved. - **11:05 a.m.** Support receives a list of affected customers and begins proactive outreach. - **2:40 p.m.** Vendor inventory reconciliation completes. Oversold items are flagged for vendor follow-up. ## Root Cause The direct cause was a change to inventory reservation behavior released the previous evening. Before the release, checkout used optimistic inventory updates: it verified availability and decremented inventory within a short transaction. The new implementation created a reservation when a customer entered checkout. To prevent two carts from reserving the same item, it selected inventory rows using `SELECT ... FOR UPDATE`. The reservation transaction remained open while the checkout service called the tax and payment services. Under normal weekday traffic, these calls typically completed quickly enough that lock contention was limited. During the Saturday rush, payment latency increased and customers frequently abandoned checkout. Inventory locks were therefore held for up to the 90-second request timeout. Checkout requests for popular products queued behind those locks. Each waiting request retained a database connection, eventually exhausting the checkout service’s connection pool. New requests could not acquire connections and failed before reaching payment or inventory validation. ## Contributing Factors - The load test modeled total checkout volume but not the concentration of purchases on a small number of popular products. - The migration was considered backward-compatible, but the altered constraint prevented a clean application rollback. - The feature rollout completed outside peak market hours and had no automatic rollback condition tied to database lock duration. - Database alerts monitored CPU, storage, and connection usage but not row-lock wait time or blocked-session count. - Checkout autoscaling was based on request latency. Adding pods increased database connection pressure without increasing useful throughput. - External tax and payment calls occurred inside the inventory transaction. - Abandoned reservation cleanup ran every five minutes, much longer than the safe lock-holding interval. - The runbook documented connection-pool exhaustion but did not include queries for identifying lock chains. ## Detection Automated monitoring detected elevated checkout latency two minutes after the incident began and elevated error rates two minutes later. The alerts correctly identified customer-facing degradation but did not identify lock contention as the underlying cause. Support reports provided the first indication that some customers saw errors after payment authorization. Payment reconciliation was initiated 73 minutes after the first failure, later than desired. ## Resolution and Recovery The team restored checkout by: 1. Disabling the new inventory-reservation path through a feature flag. 2. Pausing nonessential inventory reconciliation jobs. 3. Terminating blocked database sessions in controlled batches. 4. Returning checkout to the previous short-transaction inventory update. 5. Reconciling payment authorizations against created orders. 6. Rebuilding vendor inventory counts from completed orders and adjustments. No manual refunds were required for duplicate charges. Customers who were charged after seeing an error received order confirmations and an explanation. Vendors with oversold items were given affected order details and customer-contact guidance. ## What Went Well - Latency and error-rate alerts fired within four minutes of customer impact. - The inventory reservation was protected by a feature flag, allowing the team to disable it without a new deployment. - Payment idempotency keys prevented duplicate charges during customer retries. - Engineering, support, payments, and marketplace operations coordinated in a single incident channel. - Order and payment records contained enough identifiers to complete reconciliation without vendor intervention. ## What Went Poorly - The rollback path had not been tested after the database migration. - Load testing did not reproduce contention on popular inventory rows. - Autoscaling amplified database pressure. - Lock-specific telemetry was missing from dashboards and alerts. - The first customer-facing status update was published 34 minutes after the incident began. - Vendors saw delayed inventory without an in-product warning. - The checkout transaction included external network calls. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---| | Move tax and payment calls outside the inventory-locking transaction | Checkout Team | P0 | August 7, 2026 | | Replace long-lived row locks with atomic, conditional inventory updates | Marketplace Platform | P0 | August 14, 2026 | | Add database lock-wait and blocked-session alerts | Database Reliability | P0 | August 5, 2026 | | Cap checkout database connections independently of pod count | Site Reliability Engineering | P0 | August 4, 2026 | | Add automatic feature rollback based on checkout success and lock duration | Release Engineering | P1 | August 21, 2026 | | Create a load-test scenario for highly concentrated product demand | Performance Engineering | P1 | August 18, 2026 | | Require rollback validation for database migrations before production rollout | Database Reliability | P1 | August 12, 2026 | | Update the checkout runbook with lock-chain diagnostics and mitigation steps | Checkout Team | P1 | August 3, 2026 | | Alert on payment authorization without a customer-visible success response | Payments Team | P1 | August 20, 2026 | | Add vendor-dashboard warnings when inventory updates are delayed | Vendor Experience | P2 | September 4, 2026 | | Reduce the initial customer status-update target from 30 minutes to 15 minutes | Incident Management | P2 | August 10, 2026 | ## Preventing Recurrence The reservation feature will remain disabled until inventory changes use short, bounded transactions and the revised design passes contention-focused load testing. Future inventory releases must demonstrate safe rollback across database schema versions and include automated rollback thresholds for lock duration, connection saturation, checkout latency, and checkout success rate. We will review completion of all P0 and P1 actions in the monthly reliability meeting and publish progress to engineering and marketplace operations.