# Postmortem: Checkout Outage During Saturday Market Rush

**Date:** 2026-07-11  
**Duration:** 48 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

From 8:12 AM to 9:00 AM ET, customers were unable to complete checkout on the farmers-market marketplace. Browsing stalls and adding items to baskets continued to work, but payment confirmation failed for most orders.

The outage occurred during the peak pre-market ordering window, when customers reserve produce, baked goods, dairy, and flowers for pickup later that morning.

## Customer Impact

- 72% of checkout attempts failed during the incident.
- 1,184 customers saw payment or order confirmation errors.
- 417 completed payment authorizations were voided automatically.
- 63 vendor inventories were temporarily inaccurate due to reserved baskets not expiring on schedule.
- No customers were charged without an order being created.
- Vendors received delayed order counts for the first pickup window.

## Timeline

All times in ET.

| Time | Event |
|---|---|
| 8:12 AM | Error rate on `POST /checkout/confirm` rises above 40%. |
| 8:14 AM | On-call engineer receives checkout failure alert. |
| 8:17 AM | Support reports multiple customer complaints about failed payments. |
| 8:21 AM | Engineering identifies elevated database CPU and lock waits on basket reservation rows. |
| 8:28 AM | Recent reservation-expiration deployment is rolled back. Error rate remains elevated. |
| 8:35 AM | Team discovers background reservation cleanup job is running full-table scans. |
| 8:41 AM | Cleanup job is disabled and database lock contention begins to drop. |
| 8:47 AM | Checkout success rate recovers to 92%. |
| 8:55 AM | Expired baskets are manually cleared in batches. |
| 9:00 AM | Checkout success rate returns to normal. Incident resolved. |
| 10:15 AM | Vendor dashboards are reconciled and corrected. |
| 11:30 AM | Customer support sends apology and retry instructions to affected customers. |

## Root Cause

A change to basket reservation expiration introduced a new query that selected expired reservations by `expires_at` and `market_id`. The production `basket_reservations` table did not have a matching composite index.

During the Saturday rush, the cleanup job scanned the full reservations table every minute. This caused high database CPU and row lock contention with checkout confirmation, which also updates reservation rows to mark inventory as sold.

Checkout requests timed out while waiting for database locks, causing payment confirmation to fail.

## Contributing Factors

- The migration added application logic but did not include the required database index.
- Load testing covered normal checkout volume but not concurrent cleanup activity during peak ordering.
- The cleanup job had no circuit breaker or automatic backoff when database latency increased.
- Checkout and reservation cleanup shared the same primary database connection pool.
- Alerting detected checkout errors quickly, but there was no alert specifically for reservation lock contention.

## What Went Well

- Payment idempotency prevented duplicate charges.
- Rollback procedures were clear and executed quickly.
- Support and engineering coordinated customer messaging during the incident.
- Vendor inventory reconciliation completed before most pickups began.

## What Could Have Gone Better

- The database migration review should have caught the missing index.
- The cleanup job should have been rate-limited before production release.
- The incident took too long to distinguish between payment-provider errors and internal database contention.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---|---|
| Add composite index on `basket_reservations(market_id, expires_at)` | Database Team | 2026-07-13 | Complete |
| Add query plan review to migration checklist | Platform Team | 2026-07-17 | Open |
| Move reservation cleanup jobs to a separate worker connection pool | Marketplace Team | 2026-07-24 | Open |
| Add lock-wait and slow-query alerts for checkout tables | SRE | 2026-07-18 | Open |
| Add peak-rush load test including cleanup jobs and vendor inventory updates | QA | 2026-07-31 | Open |
| Add automatic backoff for cleanup jobs when database latency exceeds threshold | Marketplace Team | 2026-07-26 | Open |
| Create customer-support playbook for failed checkout during market rush | Support Ops | 2026-07-19 | Open |

## Prevention

Future marketplace changes that modify checkout, basket reservations, or vendor inventory counts must include:

- Database query plan review for new or changed queries.
- Load testing with scheduled jobs enabled.
- Rollback plans for both application code and background workers.
- Explicit monitoring for checkout latency, reservation lock waits, and stale basket counts.