# Postmortem: Subscription Renewal and Order Processing Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Systems affected:** Subscription renewals, order creation, customer portal  
**Status:** Resolved

## Summary

On July 24, our subscription renewal service stopped processing renewals after a database migration introduced a lock contention issue on the `subscriptions` table. Renewal workers accumulated until the queue reached capacity, delaying order creation and causing intermittent errors in the customer portal.

We restored service by rolling back the migration, terminating blocked database sessions, and gradually draining the renewal queue. No orders were lost or duplicated, but 8,412 subscription renewals were delayed.

## Customer Impact

Between 09:18 and 12:05 EDT:

- 8,412 subscription renewals were delayed by up to 4 hours.
- 1,936 customers encountered errors while viewing or modifying their subscriptions.
- 684 customers received order confirmation emails later than expected.
- 217 customers submitted duplicate update requests after seeing timeouts; these requests were deduplicated before fulfillment.
- No customers were charged more than once.
- No shipments missed the daily fulfillment cutoff.

Customer support received 326 incident-related conversations, primarily about missing renewal confirmations and an inability to change roast or delivery frequency.

## Timeline

All times are in EDT.

- **08:52** — Database migration begins as part of the scheduled application release.
- **09:07** — Migration completes successfully according to the deployment pipeline.
- **09:18** — Renewal processing latency begins increasing.
- **09:24** — Database connection utilization exceeds 90%.
- **09:31** — The first customer portal requests time out.
- **09:36** — Queue-depth alert fires, but is classified as a warning and routed only to the team chat channel.
- **09:48** — General API error-rate alert pages the on-call engineer.
- **09:55** — Incident declared SEV-1; engineering and customer support leads join the incident channel.
- **10:08** — Investigation identifies long-running transactions involving the `subscriptions` table.
- **10:21** — The team pauses renewal workers to prevent further database pressure.
- **10:34** — Engineers identify the new foreign-key validation query as the source of lock contention.
- **10:46** — Decision made to roll back the migration.
- **11:02** — Migration rollback completes.
- **11:11** — Blocked sessions are terminated and database connection utilization begins returning to normal.
- **11:23** — Customer portal error rate returns to baseline.
- **11:31** — Renewal workers restart at 10% capacity.
- **11:46** — Worker capacity increased to 50%; no renewed database contention observed.
- **12:05** — Service fully restored and incident marked mitigated.
- **14:09** — Renewal backlog fully processed.
- **15:20** — Reconciliation confirms no missing or duplicate charges or orders.

## Root Cause

The release added a foreign-key constraint from `subscription_preferences.subscription_id` to `subscriptions.id`. The migration created the constraint and immediately validated all existing rows in a single transaction.

Although the migration passed in staging, the production dataset was substantially larger and had a higher rate of concurrent writes. Constraint validation held a lock that conflicted with updates made by renewal workers. Those workers remained connected while waiting for the lock, exhausting the application's database connection pool.

Once the pool was saturated, unrelated requests from the customer portal could not obtain database connections and began timing out. Automatic worker retries increased the number of pending transactions and amplified the contention.

## Contributing Factors

- The migration was tested with production-like schema but not production-scale data or write volume.
- The constraint was validated immediately instead of being created without validation and validated separately.
- Renewal workers used fixed-interval retries without jitter or a retry limit for database lock timeouts.
- The queue-depth alert was configured as a warning and did not page the on-call engineer.
- The connection pool was shared by background workers and customer-facing API processes.
- The deployment pipeline considered migration completion sufficient and did not evaluate post-deployment lock waits or renewal throughput.

## Resolution and Recovery

We rolled back the migration, stopped renewal workers, and terminated database sessions blocked by the validation transaction. After database utilization stabilized, we restarted the workers at reduced capacity and increased throughput incrementally while monitoring lock waits, connection usage, and order creation.

We then reconciled payment-provider transactions against orders and renewal records. This confirmed that all successful charges had exactly one corresponding order and that failed renewal attempts had not resulted in charges.

## What Went Well

- Payment idempotency prevented duplicate charges during retries.
- Order reconciliation tooling allowed us to verify data integrity within three hours.
- Customer support joined the incident early and used a consistent response template.
- Gradual worker recovery prevented a second spike in database utilization.
- The fulfillment cutoff provided enough buffer to ship all affected orders on schedule.

## What Went Poorly

- Detection took 30 minutes because the earliest relevant alert did not page anyone.
- The migration pipeline did not flag the long-running validation transaction.
- Customer-facing traffic and background processing competed for the same database connections.
- The initial status page update was published 52 minutes after customer impact began.
- We lacked a documented runbook for pausing and safely restarting renewal workers.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---:|
| Require large foreign-key constraints to be created without validation and validated in a separate deployment | Database Team | P0 | August 7, 2026 |
| Add migration linting for blocking operations on high-traffic tables | Platform Team | P0 | August 14, 2026 |
| Page on sustained renewal queue growth and processing-latency breaches | SRE | P0 | August 5, 2026 |
| Add exponential backoff, jitter, and retry limits to renewal workers | Subscriptions Team | P1 | August 12, 2026 |
| Isolate worker and customer-facing API database connection pools | Platform Team | P1 | August 21, 2026 |
| Build a production-scale migration test dataset and write-load simulation | Database Team | P1 | August 28, 2026 |
| Monitor database lock waits and blocked transaction age during deployments | SRE | P1 | August 14, 2026 |
| Add renewal throughput and backlog health checks to deployment verification | Subscriptions Team | P1 | August 18, 2026 |
| Document and test the renewal worker pause-and-recovery runbook | Subscriptions Team | P2 | August 11, 2026 |
| Update incident procedures to require a status page post within 20 minutes of confirmed customer impact | Support Operations | P2 | August 7, 2026 |

## Preventing Recurrence

Future migrations affecting high-write tables will require a database review, production-scale performance testing, an explicit lock-risk assessment, and a rollback plan. Deployment health checks will include business-level signals such as renewal throughput, not only application error rates.

We will also separate background processing from customer-facing connection capacity so that a stalled batch workload cannot exhaust resources required by the customer portal.