# Postmortem: Checkout and Roast Scheduler Outage

**Incident date:** 2026-06-18  
**Duration:** 2 hours 17 minutes  
**Status:** Resolved  
**Severity:** SEV-2  
**Services affected:** Online checkout, inventory reservation, roast scheduler

## Summary

On June 18, customers were unable to complete online orders for several small-batch roasted teas. Checkout returned errors after payment authorization but before order confirmation. At the same time, the internal roast scheduler stopped assigning queued orders to production batches.

The outage was caused by a database migration that made `batch_id` required on order line items before all checkout paths had been updated to assign one. Orders containing teas that were not yet roasted failed during checkout, because those items are normally assigned to a roast batch asynchronously.

## Timeline

All times are Eastern Time.

| Time | Event |
| --- | --- |
| 09:42 | Deployment `web-2026.06.18.3` completed. |
| 09:47 | First checkout failure recorded for a preorder item. |
| 09:55 | Customer support received two reports of cards being authorized but no confirmation email being sent. |
| 10:03 | Error rate alert fired for `/checkout/complete`. |
| 10:08 | Engineering began investigation. Initial suspicion was payment provider latency. |
| 10:21 | Payment provider ruled out. Authorizations were succeeding. |
| 10:34 | Logs showed repeated database constraint failures on `order_line_items.batch_id`. |
| 10:41 | Roast scheduler errors confirmed for unassigned preorder inventory. |
| 10:52 | Deployment was rolled back. Checkout recovered for in-stock teas but preorder items still failed due to migrated schema state. |
| 11:18 | Hotfix migration allowed `batch_id` to be nullable until roast assignment completed. |
| 11:37 | Checkout success rate returned to normal. |
| 11:44 | Roast scheduler backlog drained. |
| 11:59 | Incident closed after manual review of failed payment authorizations and abandoned carts. |

## Customer Impact

- Checkout failed for 53 attempted orders.
- 31 customers saw an error after payment authorization.
- No customers were charged without an order being created. All unmatched authorizations were voided.
- 18 customers abandoned carts and did not return during the incident window.
- 14 wholesale orders were delayed from the morning roast planning run to the afternoon run.
- No incorrect tea batches were shipped.

## Root Cause

The root cause was an unsafe database migration.

The migration changed `order_line_items.batch_id` from nullable to non-nullable. This matched the desired final model for roasted inventory, but it did not account for preorder and made-to-order teas. Those items are intentionally accepted at checkout before a roast batch exists, then assigned later by the roast scheduler.

The application deployment included the schema migration, but the corresponding checkout code only assigned `batch_id` for teas already in finished inventory. When customers ordered preorder lots, the application attempted to insert line items without `batch_id`, causing the transaction to fail.

The rollback restored application code but did not reverse the database constraint, so preorder checkout continued failing until a corrective migration was applied.

## Contributing Factors

- The migration was deployed in the same release as application changes, with no expand-and-contract rollout.
- Automated tests covered in-stock checkout but not preorder checkout.
- The staging database did not include realistic preorder inventory.
- The alert identified checkout errors but did not distinguish payment failures from post-payment order creation failures.
- The rollback procedure did not include validation of schema changes.

## What Went Well

- Error rate alerting detected the incident within 16 minutes.
- Payment logs made it possible to confirm that customers were not charged incorrectly.
- Support quickly identified the customer-facing symptom.
- The roast scheduler backlog recovered automatically after the schema was fixed.

## What Went Poorly

- The initial investigation focused on the payment provider, delaying diagnosis.
- The rollback was incomplete because the schema migration remained active.
- Preorder inventory was not represented in release testing.
- Customer-facing messaging was generic and did not explain that payment authorizations would be voided.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add preorder checkout coverage to automated tests. | Engineering | 2026-06-25 | Open |
| Add realistic preorder and wholesale data to staging. | Engineering | 2026-06-27 | Open |
| Adopt expand-and-contract migrations for required column changes. | Engineering | 2026-07-02 | Open |
| Update rollback checklist to include schema validation. | Engineering | 2026-06-24 | Open |
| Split checkout alerts into payment, order creation, and confirmation email failures. | Platform | 2026-07-05 | Open |
| Add admin report for unmatched payment authorizations. | Engineering | 2026-07-08 | Open |
| Improve checkout error copy for authorization-only failures. | Support / Product | 2026-06-28 | Open |

## Prevention

Future schema changes that tighten constraints must be released in phases:

1. Add the new field or constraint-compatible code path.
2. Backfill existing and transitional records.
3. Verify production writes are compliant.
4. Enforce the constraint in a later release.
5. Include rollback steps for both code and schema state.

No migration that changes checkout-critical constraints should be deployed without test coverage for in-stock, preorder, wholesale, and subscription orders.