# Postmortem: Checkout Outage During Saturday Market Rush

Date: 2026-07-06  
Incident window: 2026-07-04 08:12-10:03 ET  
Severity: SEV-1  
Status: Resolved

## Summary

On Saturday morning, customers were unable to complete purchases for market pickup and local delivery orders. Product browsing and cart creation continued to work, but checkout failed for most users after inventory reservation. The outage occurred during peak weekly traffic for farmers and shoppers.

The incident was caused by a database connection pool exhaustion in the checkout service after a new inventory-reservation query was deployed without an index. Slow transactions held connections open, causing checkout requests to queue and eventually time out.

## Customer Impact

- 71% of checkout attempts failed during the incident window.
- 1,842 shoppers were affected.
- 327 paid orders were delayed in confirmation for up to 38 minutes.
- 94 vendors had inventory temporarily reserved for carts that never completed.
- Estimated lost gross merchandise value: $48,600.
- Customer support received 216 tickets and 43 vendor calls.
- No duplicate charges were created.

## Timeline

All times are ET.

| Time | Event |
| --- | --- |
| 08:12 | Checkout error rate begins increasing after normal Saturday traffic ramp-up. |
| 08:17 | Monitoring alerts on elevated `checkout-api` p95 latency. |
| 08:20 | On-call engineer acknowledges alert and begins investigation. |
| 08:27 | Support reports shoppers seeing "Unable to reserve items" and "Payment could not be completed" errors. |
| 08:34 | Database dashboard shows checkout service using all available connections. |
| 08:41 | Engineers identify slow inventory reservation queries introduced in the previous evening's deployment. |
| 08:49 | Checkout deploy is rolled back. Error rate decreases briefly but remains elevated due to blocked transactions. |
| 09:02 | Stale inventory reservation jobs are paused to reduce database pressure. |
| 09:14 | Database connections begin recovering. |
| 09:22 | Manual cleanup starts for expired cart reservations. |
| 09:37 | Checkout success rate returns above 95%. |
| 10:03 | Incident resolved. Backlog cleared and monitoring remains stable. |
| 11:20 | Vendor support sends affected sellers a summary and reconciliation plan. |

## Root Cause

A change to the checkout service added a vendor-level inventory availability check before payment authorization. The query filtered by `vendor_id`, `pickup_window_id`, and `reserved_until`, but the production database did not have a matching composite index.

During Saturday peak traffic, the query performed repeated sequential scans against the `inventory_reservations` table. These scans ran inside checkout transactions, keeping database connections open for longer than expected. As traffic increased, the checkout service exhausted its connection pool. New checkout requests waited for a connection until they timed out.

The rollback removed the slow query path, but recovery was delayed because existing transactions and expired reservations still needed to drain.

## Contributing Factors

- Load testing did not include Saturday morning traffic volume or realistic vendor inventory distribution.
- The migration review checked schema correctness but did not require an `EXPLAIN` plan for new high-volume queries.
- Checkout and inventory reservation shared the same database connection pool.
- Alerts detected latency but did not immediately identify connection pool saturation.
- The stale reservation cleanup job retried aggressively during the outage, adding more database load.

## What Went Well

- On-call response began within five minutes of the first alert.
- The previous checkout version was still deployable and rollback completed quickly.
- Payment idempotency prevented duplicate charges.
- Vendor reconciliation data was available from reservation and payment logs.

## What Went Poorly

- The rollback did not fully restore service because database pressure remained high.
- Support did not have an incident-specific status message ready for shoppers or vendors.
- We lacked a safe feature flag to disable the new reservation check without a deploy.
- The cleanup job made the outage worse before it was paused.

## Action Items

| Action | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Add composite index for `inventory_reservations(vendor_id, pickup_window_id, reserved_until)`. | Database Team | P0 | 2026-07-07 |
| Require `EXPLAIN ANALYZE` review for checkout, inventory, and payment queries. | Backend Team | P0 | 2026-07-12 |
| Split checkout and inventory cleanup into separate database connection pools. | Platform Team | P1 | 2026-07-19 |
| Add alert for checkout connection pool saturation. | SRE | P1 | 2026-07-14 |
| Add feature flag for vendor-level reservation checks. | Backend Team | P1 | 2026-07-21 |
| Update load tests to model Saturday peak traffic and large multi-vendor carts. | QA | P1 | 2026-07-26 |
| Add backoff and circuit breaker behavior to stale reservation cleanup job. | Backend Team | P1 | 2026-07-18 |
| Prepare shopper and vendor incident message templates. | Support Ops | P2 | 2026-07-16 |

## Prevention

Future checkout changes must include query-plan review, realistic load testing, and a rollback path that does not depend solely on redeployment. High-volume background jobs must also be able to back off automatically when checkout health is degraded.