# Postmortem: Subscription Checkout and Renewal Outage

**Date:** 2026-07-09  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Services affected:** Checkout API, subscription renewals, customer portal

## Summary

On July 9, customers were unable to start new coffee subscriptions, update existing plans, or process scheduled renewals for 1 hour and 42 minutes. The outage was caused by a database connection pool exhaustion triggered by a deployment that introduced an unbounded retry loop in the billing workflow.

The issue was mitigated by rolling back the deployment, draining stuck workers, and temporarily increasing database connection capacity.

## Customer Impact

- 3,842 scheduled subscription renewals were delayed.
- 617 checkout attempts failed.
- 214 customers were unable to update roast preferences, grind size, or delivery cadence.
- No customers were double-charged.
- Delayed renewals were retried successfully after recovery.
- 96 customers contacted support during the incident.

## Timeline

All times are in UTC.

| Time | Event |
|---|---|
| 14:03 | Version `checkout-api-2026.07.09.1` deployed to production. |
| 14:08 | Error rate for `POST /subscriptions` rises above baseline. |
| 14:11 | PagerDuty alert triggered for elevated checkout failures. |
| 14:14 | On-call engineer begins investigation. |
| 14:19 | Database metrics show active connections at 100% utilization. |
| 14:24 | Customer support reports customers seeing “Unable to update subscription” errors. |
| 14:31 | Billing worker queue depth exceeds 25,000 jobs. |
| 14:38 | Recent deployment identified as likely trigger. |
| 14:43 | Deployment rollback initiated. |
| 14:51 | Checkout API error rate begins dropping. |
| 15:02 | Stuck billing workers are restarted. |
| 15:18 | Database connection utilization returns below 60%. |
| 15:31 | Deferred renewal jobs begin processing successfully. |
| 15:45 | Customer portal fully operational. |
| 15:55 | Incident declared resolved. |

## Root Cause

A change to the billing workflow added retry behavior around payment provider calls. The retry loop did not enforce a maximum attempt count when the payment provider returned transient timeout responses.

Each retry opened a new database transaction to reload subscription state, but the transaction was not closed until the retry loop completed. During a spike in renewal traffic, worker processes held database connections while repeatedly retrying payment requests.

This exhausted the database connection pool, which caused checkout and subscription management requests to fail because they could not acquire database connections.

## Contributing Factors

- The retry logic was covered by unit tests but not by integration tests that included database transaction behavior.
- The billing worker and checkout API shared the same database connection pool.
- Existing alerts detected checkout failures but did not alert directly on billing retry volume.
- The deployment occurred during the daily renewal processing window.

## Resolution

We rolled back the checkout API deployment, restarted billing workers to clear held connections, and temporarily increased the database connection limit. Deferred renewal jobs were replayed after service recovery.

## What Went Well

- Automated alerts detected the issue within 8 minutes.
- Rollback was completed without additional customer impact.
- Renewal jobs were idempotent, so replaying delayed jobs did not create duplicate charges.
- Support and engineering coordination was fast once customer reports began.

## What Went Poorly

- The retry loop failure mode was not caught before production.
- Checkout traffic and background billing jobs were able to exhaust the same shared database pool.
- The deployment window overlapped with peak renewal processing.
- The customer-facing error message was generic and did not explain that no charge had been made.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---|---|
| Add maximum retry counts and exponential backoff to billing provider calls. | Payments Team | P0 | 2026-07-11 |
| Ensure database transactions are closed between retry attempts. | Payments Team | P0 | 2026-07-11 |
| Add integration tests for payment retries with database connection assertions. | Platform Team | P1 | 2026-07-17 |
| Split billing workers and checkout API into separate database connection pools. | Infrastructure Team | P1 | 2026-07-24 |
| Add alerts for retry volume, worker queue depth, and connection pool saturation. | SRE | P1 | 2026-07-18 |
| Block non-emergency deploys during renewal processing windows. | Engineering Managers | P2 | 2026-07-22 |
| Improve customer-facing subscription error messages. | Product Engineering | P2 | 2026-07-26 |

## Follow-Up

Engineering will review retry patterns across all payment and fulfillment workflows. The incident will also be used as a case study in the next reliability review to improve deployment timing, test coverage, and database isolation for customer-critical paths.