# Postmortem: Checkout Outage During Saturday Market Ordering

**Incident Date:** 2026-07-04  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services Affected:** Web checkout, mobile checkout, vendor order notifications

## Summary

On Saturday morning, customers were unable to complete purchases on the marketplace during peak ordering hours for participating farmers markets. Product browsing and cart creation continued to work, but checkout requests failed or timed out for many users.

The outage was caused by a database migration that added a blocking index to the `orders` table during high traffic. The migration held locks that prevented checkout writes from completing. Retries from the application increased database load and prolonged recovery.

## Customer Impact

- Approximately 38% of checkout attempts failed between 8:18 AM and 10:00 AM ET.
- 1,247 customers experienced at least one failed checkout attempt.
- 312 completed carts were abandoned during the incident window.
- 84 vendors received delayed or missing order notifications.
- Some customers placed duplicate orders after retrying checkout.
- Market coordinators at 6 locations needed manual order reconciliation before pickup windows opened.

No payment cards were charged for failed checkout attempts. Duplicate orders were refunded by Customer Support.

## Timeline

All times are Eastern Time.

| Time | Event |
| --- | --- |
| 8:05 AM | Scheduled migration begins for adding an index to `orders.market_id, pickup_date`. |
| 8:18 AM | Checkout latency rises above 10 seconds. Error rate begins increasing. |
| 8:23 AM | Monitoring alerts on elevated `POST /checkout` failures. |
| 8:27 AM | On-call engineer acknowledges alert and starts investigation. |
| 8:34 AM | Customer Support reports multiple live chat complaints about stalled checkout. |
| 8:41 AM | Engineering identifies database lock contention on the `orders` table. |
| 8:46 AM | Migration is manually cancelled. Some checkout traffic remains degraded due to queued retries. |
| 9:02 AM | Application workers are restarted to clear retry buildup and stale database connections. |
| 9:18 AM | Checkout error rate drops below 5%, but vendor notifications remain delayed. |
| 9:34 AM | Notification queue workers are scaled up to process backlog. |
| 10:00 AM | Checkout and vendor notifications return to normal operating levels. |
| 10:26 AM | Support receives duplicate-order report pattern and begins refund workflow. |
| 11:15 AM | Incident declared resolved. |
| 1:30 PM | Engineering completes initial data reconciliation report for Support and Market Operations. |

## Root Cause

A production database migration attempted to create a new index on the `orders` table without using the non-blocking migration path required for large tables.

The migration acquired locks that conflicted with checkout writes. During normal weekday traffic this may have completed quickly, but it ran during the highest weekly ordering window. As checkout requests slowed down, application-level retries increased the number of pending writes, which amplified database pressure and delayed recovery even after the migration was cancelled.

## Contributing Factors

- The migration was scheduled based on deployment availability rather than market traffic patterns.
- Our migration review checklist did not explicitly require non-blocking index creation for high-volume tables.
- Checkout retry behavior used a fixed retry count without backoff or lock-specific circuit breaking.
- Alerts detected checkout failure rate, but not database lock wait time.
- Vendor notification processing depended on successful order creation events and had limited backlog visibility.

## Resolution

Engineering cancelled the migration, restarted checkout workers to clear retry buildup, and scaled notification workers to process delayed vendor notifications. Support reconciled duplicate orders and issued refunds where needed.

The index was later created using a concurrent migration during a low-traffic maintenance window.

## What Went Well

- Checkout failure alerts triggered within minutes of customer impact.
- Customer Support quickly connected user reports to the active incident.
- Cancelling the migration reduced the main source of database contention.
- No failed checkout attempts resulted in payment capture.

## What Could Have Gone Better

- The migration should have been blocked before deployment.
- We lacked direct alerting for database lock waits.
- Retry behavior made the incident worse under contention.
- Vendor-facing status messaging was manual and delayed.
- Duplicate-order detection required ad hoc queries during the incident.

## Action Items

| Action | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Require concurrent index creation for large production tables | Platform Engineering | P0 | 2026-07-11 |
| Add automated migration linting for blocking operations | Platform Engineering | P0 | 2026-07-18 |
| Block production migrations during peak market ordering windows | Release Engineering | P0 | 2026-07-11 |
| Add database lock wait alerts and dashboard panels | SRE | P1 | 2026-07-18 |
| Add exponential backoff and circuit breaking to checkout retries | Marketplace Engineering | P1 | 2026-07-25 |
| Improve duplicate-order detection and automatic refund queueing | Payments Engineering | P1 | 2026-08-01 |
| Add vendor notification backlog alerts | Marketplace Engineering | P2 | 2026-08-01 |
| Create customer and vendor incident message templates | Support Operations | P2 | 2026-07-25 |

## Follow-Up

A review of all pending database migrations will be completed before the next Saturday market window. Any migration touching checkout, orders, payments, inventory, or vendor notifications will require explicit approval from Engineering and Market Operations before release.