# Postmortem: Checkout Failures During Saturday Market Rush

**Incident date:** July 11, 2026  
**Duration:** 8:47–10:16 a.m. EDT  
**Severity:** SEV-1  
**Services affected:** Marketplace checkout, inventory reservations, vendor order notifications  
**Status:** Resolved

## Summary

During the Saturday morning ordering peak, customers were intermittently unable to complete purchases from farmers and market vendors. A database migration introduced row-level locking on inventory records, and a retry policy in the checkout service amplified the resulting contention.

For 89 minutes, 38% of checkout attempts failed or timed out. Existing orders and vendor payouts were not affected. We restored service by disabling automatic retries, terminating blocked database sessions, and rolling back the migration.

## Customer Impact

Between 8:47 and 10:16 a.m. EDT:

- 4,812 checkout attempts were made.
- 1,829 attempts failed or timed out.
- 1,274 unique customers were affected.
- 96 customers were charged but initially saw an error page; all corresponding orders were successfully recorded.
- 312 inventory reservations remained active after failed checkouts, temporarily making products appear sold out.
- Vendor order notifications were delayed by up to 42 minutes.
- No duplicate charges, lost orders, or incorrect payouts were identified.

Customer support received 184 messages, primarily concerning checkout errors, unavailable products, and uncertainty about whether payment had succeeded.

## Timeline

All times are in EDT.

- **8:42 a.m.** — Database migration `20260711_add_inventory_audit_trigger` completed.
- **8:47 a.m.** — Checkout latency began increasing as market traffic approached its weekly peak.
- **8:51 a.m.** — The first checkout timeout was recorded.
- **8:55 a.m.** — Error-rate monitoring detected elevated HTTP 500 responses but remained below the paging threshold.
- **9:02 a.m.** — Support reported multiple customers unable to complete purchases.
- **9:05 a.m.** — The on-call engineer began investigating checkout-service logs.
- **9:11 a.m.** — The checkout error rate exceeded 20%, triggering a SEV-1 page.
- **9:17 a.m.** — Responders identified database lock waits on inventory rows.
- **9:25 a.m.** — New application instances were added. This increased database concurrency and worsened contention.
- **9:31 a.m.** — The team stopped scaling and disabled automatic checkout retries.
- **9:39 a.m.** — Blocked database sessions were terminated; latency improved temporarily.
- **9:46 a.m.** — Engineers correlated the lock waits with the morning inventory-audit migration.
- **9:53 a.m.** — The audit trigger was disabled.
- **10:02 a.m.** — Checkout success rates began returning to normal.
- **10:09 a.m.** — Stale inventory reservations were released.
- **10:16 a.m.** — Error rates and latency returned to baseline; the incident was marked mitigated.
- **10:58 a.m.** — Delayed vendor notifications finished processing.
- **1:30 p.m.** — Payment and order reconciliation confirmed that no duplicate charges or lost orders had occurred.

## Root Cause

The incident was caused by a newly deployed database trigger intended to record inventory adjustments for vendor reporting.

Checkout updates inventory in a transaction that touches several rows. The trigger synchronously inserted audit records and queried the related product row to populate metadata. Under normal traffic, the additional query added little latency. During the Saturday peak, however, concurrent checkouts frequently targeted the same limited-stock products.

These transactions acquired locks in different orders:

1. Checkout acquired an inventory-row lock and then updated the product.
2. The audit trigger queried the product before inserting its audit record.
3. Other product-management operations acquired the product-row lock before updating inventory.

This produced long lock waits and occasional deadlocks. The checkout service treated database lock errors as transient and retried the entire transaction up to three times without backoff or jitter. Retries increased the number of competing transactions, exhausting the database connection pool and causing otherwise unrelated checkouts to time out.

## Contributing Factors

- The migration was deployed 47 minutes before the marketplace’s busiest weekly period.
- Load tests used evenly distributed product selections and did not simulate many customers purchasing the same limited-stock items.
- The trigger performed a synchronous lookup that was not identified during review.
- Database lock-wait metrics were available but did not have paging alerts.
- Checkout retries had no exponential backoff or concurrency limit.
- Autoscaling responded to application latency by adding instances, increasing pressure on the database.
- The checkout alert required both a high error rate and a minimum request volume, delaying the page by approximately 16 minutes.
- The migration rollback procedure removed the trigger definition but did not explicitly terminate already blocked transactions.

## Resolution and Recovery

We disabled the checkout retry policy, terminated blocked database sessions, and disabled the inventory-audit trigger. Once lock contention fell, the connection pool recovered and checkout latency returned to normal.

We then released inventory reservations associated with failed checkouts and replayed delayed vendor notifications. Finally, we reconciled payment-provider transactions against marketplace orders to confirm that every successful charge had a corresponding order and that no customer had been charged twice.

## What Went Well

- Support escalated the customer reports quickly.
- Database lock inspection identified the immediate failure mode within 12 minutes of the SEV-1 page.
- Payment idempotency prevented duplicate charges during transaction retries.
- Existing reconciliation tooling made it possible to verify all ambiguous payment outcomes the same day.
- Vendor notifications were durable and could be replayed without data loss.

## What Went Poorly

- The initial scaling response increased database contention.
- The paging alert fired after customers and support had already detected the issue.
- The migration was approved without a production-like contention test.
- Customers who received an error after a successful charge were not shown a reliable order status.
- Incident responders lacked a documented procedure for diagnosing database lock amplification.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Replace the synchronous inventory-audit trigger with an asynchronous event-based audit pipeline | Marketplace Platform | P0 | July 24, 2026 |
| Add exponential backoff, jitter, and a retry budget to checkout database retries | Checkout Team | P0 | July 20, 2026 |
| Add paging alerts for database lock-wait duration, deadlocks, and connection-pool saturation | Reliability Engineering | P0 | July 18, 2026 |
| Prevent routine database migrations during Friday evening and Saturday market hours | Engineering Operations | P0 | July 17, 2026 |
| Add a hot-product contention scenario to checkout load tests | Performance Engineering | P1 | July 31, 2026 |
| Update autoscaling rules to account for database saturation before adding application instances | Reliability Engineering | P1 | August 7, 2026 |
| Show customers a payment-status recovery page when checkout results are ambiguous | Checkout Team | P1 | August 14, 2026 |
| Add automated cleanup for expired inventory reservations | Inventory Team | P1 | July 28, 2026 |
| Create and rehearse a database lock-contention incident runbook | Database Engineering | P1 | July 25, 2026 |
| Require query plans and lock-order analysis for triggers introduced on checkout tables | Database Engineering | P2 | August 7, 2026 |

## Lessons Learned

Database changes that are inexpensive for individual transactions can become unsafe under concentrated demand. Our tests reflected total traffic volume but not the real behavior of a farmers-market marketplace, where many buyers compete for a small number of seasonal products at the same time.

We also learned that retry and autoscaling mechanisms must be evaluated as part of the complete system. Both behaved as configured, but together they amplified pressure on the constrained dependency. Future checkout changes will be tested under skewed inventory demand, database failure conditions, and retry storms before production deployment.