# Postmortem: Saturday Checkout Outage

**Date:** July 4, 2026  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** Web checkout, mobile checkout, vendor order dashboard, order confirmation emails

## Summary

During peak Saturday morning traffic, customers were unable to reliably place orders for pickup and local delivery from farmers-market vendors. Checkout requests either timed out or returned inventory errors after payment authorization. The outage was caused by a database migration that added a foreign key constraint to the `order_items` table without using an online migration strategy. The migration held locks during a period of high write volume, causing checkout workers to stall and retry aggressively.

## Customer Impact

- 38% of checkout attempts failed between 9:18 AM and 11:00 AM EDT.
- 1,284 customers experienced failed or delayed checkout.
- 412 payment authorizations were created without completed orders; all were automatically voided within 3 hours.
- 96 vendor dashboards showed stale order counts.
- 73 vendors received orders later than expected, affecting packing windows for same-day pickup.
- Customer support received 219 tickets related to failed checkout, duplicate payment holds, or missing confirmation emails.

No completed orders were lost.

## Timeline

All times are in EDT.

| Time | Event |
| --- | --- |
| 9:12 AM | Database migration `20260704_add_order_items_vendor_fk` starts automatically as part of the morning deploy. |
| 9:18 AM | Checkout latency rises above 8 seconds. First timeout errors appear in application logs. |
| 9:22 AM | Error budget alert fires for checkout API 5xx rate. On-call engineer is paged. |
| 9:26 AM | Vendor dashboard queue backlog begins increasing due to delayed order creation events. |
| 9:31 AM | On-call identifies elevated lock waits on the primary PostgreSQL database. |
| 9:37 AM | Morning deploy is paused. Application rollback is started but does not resolve lock contention. |
| 9:44 AM | Migration process is found still running and holding locks on `order_items`. |
| 9:47 AM | Migration is cancelled. Lock waits drop, but checkout workers remain saturated by retries. |
| 9:55 AM | Checkout worker autoscaling reaches maximum capacity. Queue depth continues to grow. |
| 10:06 AM | Retry policy for checkout finalization is temporarily reduced from 5 attempts to 1 attempt. |
| 10:18 AM | Database CPU and connection usage return to normal. Checkout success rate begins recovering. |
| 10:34 AM | Vendor dashboard event backlog starts draining. |
| 10:51 AM | Checkout success rate returns above 99%. |
| 11:00 AM | Incident is declared resolved. |
| 12:14 PM | Payment authorization reconciliation job completes. No duplicate charges are found. |
| 2:03 PM | Customer support receives final list of affected customers for outreach. |

## Root Cause

The direct cause was a blocking database migration on the `order_items` table during peak marketplace traffic.

The migration added a foreign key constraint from `order_items.vendor_id` to `vendors.id`. Because the migration was executed using the default Rails migration behavior, PostgreSQL validated the constraint immediately while holding locks that blocked concurrent writes to `order_items`.

Checkout depends on writing to `orders`, `order_items`, `inventory_reservations`, and `payment_authorizations` within a single workflow. Once writes to `order_items` stalled, checkout requests exceeded their timeout. The checkout service retried failed finalization attempts, which increased database connection pressure and slowed recovery even after the migration was cancelled.

Contributing factors:

- Schema migrations were allowed to run automatically during peak marketplace hours.
- The migration review checklist did not require online-safe patterns for foreign key validation.
- Checkout retry behavior did not distinguish between transient network failures and database lock contention.
- Alerts detected checkout failure rate quickly, but there was no dedicated alert for long-running schema locks.
- The deploy system did not surface active migration lock state during rollback.

## Resolution

The migration was cancelled, the deploy was paused, and checkout retry behavior was temporarily reduced to prevent additional load on the database. Once lock contention cleared and worker queues drained, checkout recovered.

The foreign key migration will be reintroduced using an online-safe approach:

1. Add the foreign key constraint as `NOT VALID`.
2. Validate the constraint in a separate migration during a low-traffic window.
3. Monitor lock waits and cancel automatically if thresholds are exceeded.

## What Went Well

- Checkout failure alerts paged the on-call engineer within 4 minutes.
- Payment reconciliation correctly identified and voided incomplete authorizations.
- No completed orders were lost.
- Vendor order events replayed successfully after the backlog drained.

## What Did Not Go Well

- A blocking migration reached production during the busiest weekly ordering window.
- Rollback did not stop the active database migration.
- Retry behavior amplified load during the incident.
- Support did not receive an affected-customer export until after the outage was resolved.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Block automatic schema migrations during Saturday market hours | Platform | July 12, 2026 | Open |
| Add migration linter rules for foreign keys, indexes, and table rewrites | Platform | July 19, 2026 | Open |
| Update migration checklist with online PostgreSQL patterns | Engineering Enablement | July 16, 2026 | Open |
| Add alert for database lock waits over 30 seconds | SRE | July 14, 2026 | Open |
| Show active migration state in deploy and rollback UI | Developer Tools | July 26, 2026 | Open |
| Change checkout retry policy to use bounded retries with lock-aware backoff | Marketplace Engineering | July 21, 2026 | Open |
| Create automated affected-customer export for checkout incidents | Data Platform | July 24, 2026 | Open |
| Run a game day for checkout degradation and payment reconciliation | SRE + Marketplace Engineering | August 2, 2026 | Open |

## Follow-Up

A revised migration plan will be reviewed before the foreign key is reintroduced. The incident review group will verify completion of all SEV-1 action items before the next Saturday peak period.