# Postmortem: Subscription Checkout and Roasting Queue Outage

**Date:** 2026-06-18  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-1  
**Services affected:** Subscription checkout, account portal, roasting queue scheduler, customer notifications  
**Status:** Resolved

## Summary

On June 18, 2026, customers were unable to create, modify, or pause coffee subscriptions for 1 hour and 47 minutes. Existing subscriptions scheduled during the incident window were also delayed from entering the roasting queue.

The outage was triggered by a database migration that added a non-null column with a default value to the `subscriptions` table. The migration caused a long-running table rewrite and held locks on a high-traffic table. Application requests began timing out, background workers retried failed jobs aggressively, and database connection pool exhaustion spread the issue to adjacent services.

## Customer Impact

- 3,842 customers saw errors when attempting to manage subscriptions.
- 617 checkout attempts failed before payment authorization.
- 289 subscription renewals were delayed by up to 3 hours.
- 94 customers received delayed shipment confirmation emails.
- No duplicate charges occurred.
- No subscription data was lost.

Customer-facing symptoms included:

- Checkout failures after selecting coffee frequency.
- Account portal timeouts when changing grind size, roast preference, or delivery date.
- Delayed renewal confirmations.
- Support chat volume increased by approximately 4.7x during the incident.

## Timeline

All times are in Eastern Time.

| Time | Event |
| --- | --- |
| 09:12 | Migration `20260618_add_fulfillment_region_to_subscriptions` starts during normal deploy. |
| 09:15 | API latency for `/subscriptions` rises above 2 seconds. |
| 09:18 | First checkout errors reported by frontend monitoring. |
| 09:21 | Background renewal workers begin retrying failed subscription updates. |
| 09:24 | Database CPU reaches 92%; active connections hit pool limit. |
| 09:27 | On-call engineer receives SEV-1 alert for elevated 5xx rate. |
| 09:31 | Deploy is paused. Initial assumption is a bad application release. |
| 09:39 | Rollback of application release completes, but errors continue. |
| 09:44 | Investigation identifies blocked queries on the `subscriptions` table. |
| 09:49 | Migration process is terminated. Locks begin clearing. |
| 09:56 | API error rate starts dropping but workers continue generating retry load. |
| 10:03 | Renewal workers are paused manually. |
| 10:17 | Database connections and query latency return to normal levels. |
| 10:31 | Subscription checkout is confirmed healthy. |
| 10:42 | Renewal workers are resumed at reduced concurrency. |
| 10:59 | Backlog is drained. Incident is resolved. |

## Root Cause

The root cause was an unsafe schema migration on the `subscriptions` table:

```sql
ALTER TABLE subscriptions
ADD COLUMN fulfillment_region text NOT NULL DEFAULT 'east';
```

Because the table contained approximately 18 million rows, the database rewrote the table while holding locks that blocked reads and writes required by the subscription API. This caused request timeouts across checkout and account management flows.

The retry behavior of renewal workers amplified the outage. Failed jobs were retried immediately with insufficient jitter and no global rate limit, increasing database pressure while the table was locked.

## Contributing Factors

- The migration was reviewed as a small application change rather than a high-risk database operation.
- Staging data volume was too small to reveal lock duration.
- Migration checks did not flag `NOT NULL DEFAULT` changes on large tables.
- Worker retry policy optimized for transient API failures, not database lock contention.
- Dashboards showed database saturation, but not the specific blocking migration until manual investigation.

## Resolution

The migration process was terminated, which released the table lock. Renewal workers were paused to reduce retry pressure. Once database latency normalized, workers were resumed with reduced concurrency and the renewal backlog was processed successfully.

A safer migration plan was created:

1. Add the column as nullable with no default.
2. Backfill in small batches.
3. Add the default.
4. Add the `NOT NULL` constraint after validation.

## What Went Well

- Alerts fired within 10 minutes of customer impact.
- No payments were duplicated.
- The team paused workers quickly after identifying retry amplification.
- Customer support had a status update within 30 minutes.

## What Could Be Improved

- The initial rollback focused on application code and delayed database investigation.
- Migration risk was not visible enough during code review.
- Worker retry behavior made recovery slower.
- Staging did not represent production table size or query patterns.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add automated checks for unsafe migrations on large tables | Platform | 2026-07-02 | Open |
| Require database review for migrations touching `subscriptions`, `orders`, or `payments` | Engineering Managers | 2026-07-05 | Open |
| Implement batched backfill helper with progress reporting and throttling | Platform | 2026-07-12 | Open |
| Add retry jitter and global concurrency limits to renewal workers | Subscriptions Team | 2026-07-08 | Open |
| Add dashboard panel for blocking queries and active migration process IDs | SRE | 2026-07-03 | Open |
| Increase staging subscription table volume to production-like scale | Data Engineering | 2026-07-15 | Open |
| Update incident runbook to check database locks before application rollback | SRE | 2026-07-01 | Open |

## Prevention

Future schema changes to high-traffic tables will use expand-and-contract migrations. Any migration expected to touch more than 100,000 rows must include a rollback plan, production timing estimate, and explicit database review before deployment.

Worker systems that depend on subscription writes will also use bounded retries with jitter and centralized rate limits so they do not amplify database incidents.