# Postmortem: Subscription Renewal and Order Processing Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** Subscription renewals, customer checkout, fulfillment queue

## Summary

On July 24, a database migration caused subscription renewal workers to repeatedly fail while processing active subscriptions. Automatic retries increased database load, exhausted the application connection pool, and degraded customer checkout and account management.

We stopped the retry loop, rolled back the migration, and replayed affected renewals after validating order and payment state. No customer payment details were exposed, and no confirmed orders were lost.

## Customer Impact

During the incident:

- 8,412 subscription renewals were delayed.
- 1,936 checkout attempts failed or timed out.
- 624 customers were charged later than their scheduled renewal time.
- 117 customers received duplicate payment authorization attempts; 14 temporary duplicate authorization holds appeared, but no duplicate charges settled.
- Account and subscription-management pages intermittently returned errors.
- Fulfillment exports were delayed by approximately four hours.

Affected customers received an email explaining the delay. Support agents were given a list of customers with duplicate authorization attempts for proactive follow-up.

## Detection

Our checkout error-rate alert fired 19 minutes after the migration. The renewal worker failure rate increased immediately, but the existing alert evaluated failures as a percentage of completed jobs. Because few jobs completed during the incident, the alert did not trigger as intended.

## Timeline

All times are in Eastern Daylight Time.

- **09:02** — Deployment containing the subscription schema migration begins.
- **09:08** — Migration completes successfully according to the deployment system.
- **09:09** — Renewal workers begin failing with database constraint errors.
- **09:12** — Automatic retries increase database connection usage.
- **09:18** — Checkout latency rises above the normal range.
- **09:27** — Checkout error-rate alert fires; on-call engineer begins investigation.
- **09:35** — Incident declared SEV-1. Application and database engineers join.
- **09:43** — Team identifies connection-pool exhaustion and scales the API fleet. Error rates briefly improve but then rise again.
- **10:01** — Renewal retries are identified as the primary source of database load.
- **10:08** — Renewal workers are paused. Checkout begins recovering.
- **10:19** — Team confirms the migration made a previously nullable column mandatory before all subscription records were backfilled.
- **10:31** — Rollback begins.
- **10:48** — Rollback completes; database load returns to normal.
- **11:02** — Checkout and account-management error rates return to baseline.
- **11:21** — Renewal workers resume at reduced concurrency.
- **11:49** — Incident resolved; renewal backlog processing continues under monitoring.
- **15:36** — Renewal backlog is fully processed and fulfillment exports resume.

## Root Cause

The release introduced a `NOT NULL` constraint on the `subscriptions.next_roast_profile_id` column. The migration assumed every active subscription had already received a roast-profile value through a backfill job.

The backfill, however, excluded subscriptions created through an older gift-subscription flow. When renewal workers loaded those records, writes failed against the new constraint.

The worker treated these failures as transient database errors and retried them with exponential backoff. Thousands of failing jobs repeatedly opened transactions and consumed database connections. This exhausted the shared connection pool used by renewal workers and customer-facing APIs, causing checkout and account requests to time out.

## Contributing Factors

- The migration combined a data assumption and constraint enforcement in one deployment.
- Staging did not contain representative legacy gift-subscription records.
- The backfill verification query counted only standard subscriptions.
- Permanent constraint violations were classified as retryable errors.
- Renewal workers and customer-facing APIs shared the same database connection limit.
- The migration’s success check verified schema completion but not data compatibility.
- Scaling the API fleet increased the number of potential database connections and briefly intensified contention.

## Resolution and Recovery

We paused renewal workers to stop the retry storm, then removed the new constraint. Once database connection usage stabilized, customer-facing services recovered without further changes.

Before replaying renewals, we reconciled payment-provider events against internal order records to prevent duplicate charges. We then resumed workers at reduced concurrency and monitored payment attempts, order creation, and fulfillment exports until the backlog cleared.

## What Went Well

- The checkout error-rate alert detected customer-facing degradation.
- The incident commander established clear ownership within minutes of escalation.
- Pausing workers provided immediate relief without requiring a full application rollback.
- Payment idempotency keys prevented duplicate charges from settling.
- The fulfillment team held exports until order reconciliation was complete.

## What Went Poorly

- Worker failures were not detected before customer-facing services degraded.
- Migration review did not require evidence that the backfill covered every subscription type.
- Error classification obscured a deterministic failure as a transient one.
- Shared database capacity allowed a background workload to affect checkout.
- The initial mitigation of scaling the API fleet increased database pressure.
- Support did not receive a customer-impact list until nearly two hours after resolution.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Separate schema enforcement from data backfills across deployments | Platform Engineering | P0 | August 7, 2026 |
| Add pre-migration validation that blocks constraints when incompatible rows exist | Database Engineering | P0 | August 7, 2026 |
| Classify database constraint violations as non-retryable worker failures | Subscriptions Team | P0 | August 5, 2026 |
| Add an absolute renewal-failure-count alert independent of completed jobs | Observability Team | P0 | August 3, 2026 |
| Add legacy gift subscriptions to staging fixtures and migration tests | Quality Engineering | P1 | August 14, 2026 |
| Isolate background workers in a dedicated database connection pool | Platform Engineering | P1 | August 21, 2026 |
| Add circuit breakers that pause renewal processing when failure rates exceed thresholds | Subscriptions Team | P1 | August 21, 2026 |
| Document database saturation risks in the scaling runbook | SRE | P1 | August 10, 2026 |
| Automate payment and order reconciliation reports for incident response | Payments Team | P2 | September 4, 2026 |
| Run a failure-mode exercise covering renewal backlog recovery | SRE | P2 | September 11, 2026 |

## Prevention

Future schema constraints affecting subscription processing will follow a three-stage rollout:

1. Deploy code that supports both old and new data shapes.
2. Backfill data and verify completeness across all subscription types.
3. Enforce the constraint only after automated compatibility checks pass.

We will also isolate renewal workloads from customer-facing traffic and stop retrying deterministic database errors. These changes reduce both the probability of a similar failure and the blast radius if one occurs.