# Postmortem: Subscription Renewal and Fulfillment Outage

**Incident date:** July 22, 2026  
**Duration:** 2 hours, 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 22, our subscription renewal service failed while processing the morning renewal batch. A database migration introduced a new uniqueness constraint that was incompatible with the service’s retry behavior. When payment-provider responses slowed, renewal jobs exceeded their timeout and were retried. The retries attempted to insert duplicate payment-attempt records, triggering database errors and causing the worker queue to back up.

Customers experienced delayed renewals, duplicate payment authorization holds, and late fulfillment notifications. No customers were charged twice, and no subscription or payment data was lost.

## Customer Impact

Between 08:14 and 10:31 EDT:

- 18,742 subscription renewals were delayed.
- 1,286 customers saw two temporary authorization holds on their cards.
- 614 customers received fulfillment confirmation emails between one and four hours late.
- 203 customers could not update or pause their subscriptions because the account API depended on the overloaded primary database.
- 87 support tickets were opened during the incident.

All delayed renewals were processed by 13:48 EDT. Duplicate authorization holds were voided automatically or expired according to the customer’s bank policy. No duplicate captures occurred.

## Timeline

All times are in EDT.

- **07:45** — Version `renewals-2026.07.22.1`, including a payment-attempt uniqueness constraint, is deployed.
- **08:00** — The scheduled daily renewal batch begins.
- **08:11** — Payment-provider latency rises from approximately 400 ms to 3.8 seconds.
- **08:14** — Renewal workers begin timing out and retrying jobs.
- **08:16** — Duplicate payment-attempt inserts start failing against the new constraint.
- **08:19** — Queue depth alert fires, but is classified as a warning because the threshold is based on the previous month’s batch volume.
- **08:27** — Database connection utilization exceeds 90%.
- **08:32** — Account API error rate crosses the paging threshold, and the on-call engineer is paged.
- **08:38** — The incident is declared SEV-1. Renewal deployments are paused.
- **08:46** — The team identifies constraint violations in renewal-worker logs.
- **08:55** — Worker concurrency is reduced to protect the primary database.
- **09:07** — New renewal processing is paused. Subscription management begins recovering.
- **09:18** — Engineers confirm that duplicate authorization requests were created but no duplicate captures occurred.
- **09:31** — A patch changes payment-attempt creation to an idempotent upsert and reuses the original provider idempotency key.
- **09:49** — The patched worker is deployed to 10% of the fleet.
- **10:02** — Constraint errors stop in the canary workers.
- **10:11** — The patch is deployed to all renewal workers.
- **10:31** — Error rates and database utilization return to normal. The incident is resolved.
- **13:48** — The renewal backlog is fully processed.
- **15:20** — Support receives a list of affected customers and approved response language.

## Root Cause

The immediate cause was an incompatibility between a new database constraint and the renewal worker’s retry implementation.

The deployment added a unique constraint on `(subscription_id, billing_period, attempt_number)` to prevent duplicate payment-attempt records. The worker incremented `attempt_number` only after receiving a definitive response from the payment provider. When a request timed out, the job was retried with the same attempt number, and the worker attempted a second insert. The database correctly rejected that insert.

The worker treated this expected constraint violation as a transient database failure. It retried repeatedly without recognizing that the original payment request might still have succeeded. This behavior increased database load and created new payment-provider requests with different idempotency keys, resulting in duplicate authorization holds.

Payment-provider latency exposed the defect, but the outage was caused by our retry and idempotency design.

## Contributing Factors

- The migration was tested against successful and declined payments, but not ambiguous timeouts.
- Application and database changes were deployed together without a compatibility window.
- Payment idempotency keys were generated per worker execution rather than per logical renewal attempt.
- Queue-depth alert thresholds had not been adjusted for recent subscriber growth.
- The account API and renewal workers shared the same database connection pool.
- The runbook did not document how to pause renewals without also delaying subscription-management operations.

## Resolution and Recovery

We paused new renewal processing, reduced worker concurrency, and deployed a patch that makes payment-attempt creation idempotent. Retries now load the existing attempt record and reuse its payment-provider idempotency key.

After database utilization stabilized, we resumed renewals gradually and monitored captures, authorization holds, queue depth, and fulfillment events. We reconciled all affected payment attempts against provider records before releasing orders to fulfillment.

## What Went Well

- Capture reconciliation confirmed early that no customers were charged twice.
- The renewal batch could be paused without losing queued jobs.
- The patched behavior was validated through a canary deployment before full rollout.
- Support and fulfillment teams coordinated to prevent affected orders from being shipped before payment status was confirmed.

## What Went Poorly

- The initial queue alert did not page the on-call engineer.
- Logs reported constraint violations without the subscription or logical-attempt identifier needed for quick correlation.
- Shared database capacity allowed a batch-processing failure to affect subscription management.
- Customer support learned about the incident from incoming tickets before receiving an internal notice.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Generate payment idempotency keys from the subscription and billing period | Payments | P0 | July 29, 2026 |
| Add timeout and ambiguous-response scenarios to renewal integration tests | Payments | P0 | July 29, 2026 |
| Separate renewal-worker and account-API database connection pools | Platform | P0 | August 5, 2026 |
| Add paging alerts for renewal failure rate and queue age | SRE | P0 | July 26, 2026 |
| Introduce expand-and-contract deployment rules for schema changes | Platform | P1 | August 12, 2026 |
| Add automated payment-provider reconciliation before fulfillment release | Fulfillment | P1 | August 19, 2026 |
| Update the renewal incident runbook and conduct a game day | SRE | P1 | August 14, 2026 |
| Add subscription and logical-attempt identifiers to structured logs | Payments | P1 | August 5, 2026 |
| Review capacity thresholds after each 10% increase in active subscriptions | SRE | P2 | August 21, 2026 |
| Define customer-notification criteria for payment incidents | Support | P2 | August 7, 2026 |

## Prevention

The primary preventive change is to treat a renewal as one logical payment operation across all retries. Database records and payment-provider requests must share a stable idempotency key, and ambiguous responses must be reconciled before another request is sent.

We will also isolate customer-facing database capacity from batch workloads and require failure-mode testing for migrations that change uniqueness or retry semantics.