# Postmortem: Checkout Outage During Saturday Market

**Incident date:** July 25, 2026  
**Duration:** 8:07–9:21 a.m. EDT  
**Severity:** SEV-1  
**Services affected:** Checkout, inventory reservations, vendor order notifications  
**Status:** Resolved

## Summary

On Saturday, July 25, customers were unable to complete purchases on the marketplace during the busiest ordering window before local markets opened. Checkout requests either timed out or returned an error after payment details were submitted.

The outage began after a routine deployment introduced a database query that locked inventory rows longer than expected. Saturday traffic and a vendor catalog import caused lock contention to spread across the primary database. Checkout workers exhausted their connection pools, and retries increased the load.

We restored service by disabling the new inventory validation path, pausing catalog imports, and restarting checkout workers. No customers were charged for failed orders. Some successful orders were confirmed late, which created uncertainty for customers and vendors.

## Customer Impact

From 8:07 to 9:21 a.m. EDT:

- 3,842 checkout attempts failed or timed out.
- 1,126 customers were affected.
- 417 customers retried checkout more than once.
- 286 orders were completed but confirmations were delayed by up to 38 minutes.
- 74 vendors received late order notifications.
- Estimated lost gross merchandise value was $61,400.
- Browsing, search, vendor pages, and existing order history remained available.

Payment authorization is performed only after inventory reservation succeeds, so failed checkouts did not produce charges. We verified this against payment-provider records. Twelve customers saw temporary card-verification holds from repeated attempts; those holds were released automatically by their banks.

## Timeline

All times are in Eastern Daylight Time.

- **7:42 a.m.** — Version `2026.07.25.1` was deployed to checkout workers. The release included stricter inventory validation intended to prevent overselling.
- **7:55 a.m.** — A scheduled import began updating approximately 18,000 product and inventory records for weekend vendors.
- **8:07 a.m.** — Checkout latency exceeded 10 seconds as row-lock waits increased on the primary database.
- **8:10 a.m.** — The first checkout connection pool reached capacity.
- **8:12 a.m.** — The error-rate alert fired. The on-call engineer began investigating.
- **8:16 a.m.** — Automated retries increased database traffic to roughly four times the normal Saturday peak.
- **8:21 a.m.** — Support reported multiple customers unable to place orders. The incident was declared SEV-1.
- **8:27 a.m.** — Engineers identified inventory reservation queries as the source of most blocked database sessions.
- **8:32 a.m.** — Catalog imports were paused. Database load declined slightly, but checkout remained impaired because retry queues were still growing.
- **8:39 a.m.** — The team disabled the new inventory validation path using a feature flag.
- **8:44 a.m.** — New checkout requests began succeeding, but latency remained elevated.
- **8:49 a.m.** — Checkout workers were restarted in batches to clear exhausted connection pools and stale retries.
- **8:57 a.m.** — Checkout success rate recovered above 95%.
- **9:06 a.m.** — Delayed order confirmations began draining normally.
- **9:21 a.m.** — Error rate, latency, and queue depth returned to baseline. The incident was marked resolved.
- **10:03 a.m.** — Engineers completed reconciliation of orders and payment-provider records.
- **11:18 a.m.** — Affected vendors received a summary of delayed orders.
- **1:30 p.m.** — Customer support began contacting customers whose orders succeeded after an initial timeout.

## Root Cause

The deployment changed inventory reservation from a single conditional update to a transaction that:

1. Selected every item in the cart using `SELECT ... FOR UPDATE`.
2. Loaded vendor availability rules.
3. Recalculated the cart.
4. Updated reserved quantities.
5. Committed the transaction.

The availability-rule lookup and cart recalculation occurred while inventory rows remained locked. Under normal weekday traffic, transactions completed quickly enough that the additional lock time was not visible.

During the Saturday peak, the scheduled catalog import updated many of the same inventory rows. Checkout transactions and imports acquired locks in different orders: checkout locked inventory before product metadata, while the import locked product metadata before inventory. This created long lock waits and occasional deadlocks.

Checkout workers retried database failures immediately, up to three times. Those synchronized retries consumed remaining database connections and amplified the original contention. Once connection pools were exhausted, otherwise healthy checkout requests could not reach the database.

## Contributing Factors

- The new query was load-tested against checkout traffic but not against concurrent catalog imports.
- Staging data did not reflect the number of products shared across multi-vendor carts.
- The catalog import and checkout paths acquired database locks in inconsistent orders.
- Database retries used fixed delays without jitter or a global retry budget.
- The checkout alert measured application errors but did not directly alert on lock-wait time.
- The deployment occurred shortly before the marketplace’s predictable Saturday traffic peak.
- The feature flag disabled the new validation logic, but the rollback procedure did not automatically clear queued retries or recycle exhausted workers.

## Resolution and Recovery

The incident team paused catalog imports and disabled the new inventory validation path. This stopped new long-running inventory transactions, but queued retries continued consuming database connections. Restarting checkout workers in batches cleared those retries without interrupting successful in-flight orders.

After recovery, we reconciled marketplace orders with payment-provider records, regenerated missing confirmations, and sent delayed vendor notifications. Support contacted customers whose checkout timed out even though their order was ultimately created.

## What Went Well

- The feature flag allowed the problematic validation path to be disabled without a full deployment.
- Payment authorization occurred after inventory reservation, preventing charges for failed checkouts.
- Order and payment reconciliation tools confirmed data consistency within 42 minutes of recovery.
- Support reports helped establish the scope of customer-visible failures early in the incident.
- Restarting workers in batches restored capacity without causing another traffic spike.

## What Went Poorly

- The initial alert did not identify database lock contention, adding time to diagnosis.
- Immediate retries amplified load during database degradation.
- The deployment and catalog import overlapped with the highest-risk traffic window.
- The public status page was updated 24 minutes after the incident began.
- Vendors could not distinguish delayed notifications from missing orders.
- The incident runbook did not include steps for draining retry queues.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---:|
| Move availability-rule lookup and cart recalculation outside the inventory-locking transaction | Checkout Team | P0 | August 3, 2026 |
| Enforce a consistent lock order across checkout and catalog import paths | Data Platform | P0 | August 7, 2026 |
| Replace immediate retries with exponential backoff, jitter, and a per-request retry budget | Checkout Team | P0 | August 5, 2026 |
| Add database lock-wait, deadlock-rate, and connection-pool saturation alerts | Reliability Engineering | P0 | August 4, 2026 |
| Prevent production deployments within 90 minutes of weekend market opening without incident-commander approval | Engineering Operations | P1 | August 1, 2026 |
| Reschedule large vendor imports outside peak ordering windows | Marketplace Operations | P1 | August 1, 2026 |
| Add concurrent checkout-and-import scenarios to performance testing | Quality Engineering | P1 | August 14, 2026 |
| Update staging datasets to represent multi-vendor carts and peak catalog size | Data Platform | P1 | August 14, 2026 |
| Add retry-queue draining and worker recycling procedures to the checkout runbook | Reliability Engineering | P1 | August 6, 2026 |
| Update the vendor dashboard to show notification delivery status separately from order status | Vendor Experience | P2 | September 4, 2026 |
| Automate status-page updates when checkout enters a sustained SEV-1 state | Reliability Engineering | P2 | August 21, 2026 |

## Prevention

The immediate code fix will shorten inventory transactions and standardize lock acquisition. Longer term, inventory reservations will be isolated from catalog maintenance workloads so that a bulk vendor update cannot consume checkout capacity. We will also introduce admission control for retries and catalog imports when database lock waits rise.

We will validate these changes with a production-scale load test that combines peak Saturday checkout volume, multi-vendor carts, catalog imports, and simulated database contention before re-enabling the stricter inventory validation path.