# Postmortem: Subscription Checkout and Renewal Outage

**Date:** 2026-07-08  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Services affected:** Subscription checkout, renewal billing, customer account dashboard

## Summary

On July 8, customers were unable to start new coffee subscriptions or update existing subscriptions for 1 hour and 47 minutes. During the same window, scheduled renewal jobs failed for a subset of active subscribers.

The outage was caused by a database migration that added a non-null column to the `subscriptions` table without a safe default. Application writes began failing when checkout and renewal paths attempted to create subscription records without providing the new field.

## Customer Impact

- New customers could not complete subscription checkout.
- Existing customers could not update grind size, roast preference, delivery cadence, or payment method.
- 3,842 renewal attempts failed during the outage window.
- 614 customers saw duplicate “payment failed” emails due to retry behavior in the billing worker.
- No customers were charged incorrectly.
- Failed renewals were replayed successfully after the fix, except for 27 customers whose payment methods were already invalid.

## Timeline

All times in ET.

- **09:12** - Migration deployed to production adding `origin_preference` as a non-null column on `subscriptions`.
- **09:15** - First checkout errors begin appearing in application logs.
- **09:18** - Billing renewal worker starts failing subscription creation retries.
- **09:22** - Customer support reports increased chat volume about checkout failures.
- **09:25** - On-call engineer receives alert for elevated `500` rate on checkout API.
- **09:31** - Incident declared as SEV-2.
- **09:38** - Engineering identifies failed inserts into `subscriptions` due to missing `origin_preference`.
- **09:44** - Renewal worker paused to prevent further failed retries and customer emails.
- **09:51** - Hotfix prepared to provide a default `origin_preference` value in application writes.
- **10:07** - Hotfix deployed.
- **10:12** - Checkout success rate returns to normal.
- **10:19** - Account dashboard update errors return to baseline.
- **10:31** - Renewal worker resumed with email sending disabled during replay.
- **10:59** - Failed renewals replayed successfully.
- **11:02** - Incident resolved.

## Root Cause

The immediate cause was an unsafe schema migration. The migration added a required `origin_preference` column to the `subscriptions` table before all application write paths had been updated to provide a value.

The checkout service had been updated and tested, but two other write paths were missed:

- Renewal billing worker
- Account dashboard subscription update flow

Because the column was `NOT NULL` and had no database default, inserts and some updates failed at the database layer.

## Contributing Factors

- Migration review did not require proof that all write paths had been updated.
- Staging data did not include active renewal scenarios.
- The billing worker retried failures with customer-facing emails enabled.
- Checkout alerting triggered quickly, but renewal failure alerting lagged by 12 minutes.
- The feature rollout plan combined schema enforcement and application behavior changes in the same deploy window.

## Detection

The incident was detected by an automated alert on elevated `500` responses from the checkout API. Customer support reports arrived shortly before the alert fired but were not the primary detection mechanism.

## Resolution

We deployed a hotfix that supplied a default `origin_preference` value of `balanced` across all subscription write paths. We then paused customer-facing renewal failure emails, replayed failed renewal jobs, and confirmed successful processing.

## What Went Well

- Checkout API alerting detected the issue within 10 minutes.
- The failing database constraint made the broken write path clear once logs were inspected.
- Failed renewal jobs were retained and could be replayed.
- No incorrect charges were issued.

## What Went Poorly

- The migration was not backward compatible.
- Renewal behavior was not covered by staging tests.
- Retry emails created unnecessary customer confusion.
- The incident affected both new revenue and existing customer retention workflows.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---:|---|
| Add migration checklist requiring backward-compatible schema changes | Engineering Manager | 2026-07-15 | Open |
| Update migration tooling to warn on `NOT NULL` columns without defaults | Platform | 2026-07-22 | Open |
| Add integration tests for renewal billing worker subscription writes | Billing Team | 2026-07-19 | Open |
| Add staging fixtures for active subscriptions with renewal retries | QA | 2026-07-18 | Open |
| Disable customer-facing emails during automated retry storms | Lifecycle Team | 2026-07-24 | Open |
| Add alert for renewal worker failure rate above 2% over 5 minutes | Observability | 2026-07-16 | Open |
| Document safe rollout pattern for required subscription fields | Platform | 2026-07-17 | Open |

## Prevention Plan

Future required subscription fields will be introduced in three phases:

1. Add nullable column or database default.
2. Deploy application code that writes and reads the new field.
3. Backfill existing records and enforce `NOT NULL` only after verification.

This pattern will be added to the engineering release checklist and enforced during migration review.