# Postmortem: Delayed and Failed Podcast Episode Publishing

**Incident date:** July 8, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 8, our episode publishing pipeline experienced a sustained outage that prevented newly uploaded episodes from being processed and published. Existing podcast feeds and previously published audio remained available throughout the incident.

The outage began after a routine deployment introduced a database query that performed poorly under production traffic. The resulting connection saturation caused publishing workers to stop claiming new jobs. Automated retries then increased database load, extending the incident.

We restored service by rolling back the deployment, disabling automatic retries, and restarting publishing workers in controlled batches. All delayed episodes were processed within 74 minutes of service restoration.

## Customer Impact

From 14:06 to 16:53 UTC:

- 18,742 episode publishing requests were delayed.
- 1,386 publishing requests displayed a failure state and required automatic recovery.
- 612 scheduled episodes missed their intended publication time by more than 30 minutes.
- RSS feeds for existing episodes remained available.
- Audio playback for previously published episodes was unaffected.
- Analytics ingestion continued normally, though dashboard data was delayed by up to 12 minutes due to shared database contention.

No episode files, metadata, analytics events, or customer data were lost.

Customers saw publishing jobs remain in “Processing” or change to “Failed.” Some customers retried uploads, creating duplicate draft episodes. Support received 428 incident-related conversations.

## Detection

Our database connection-pool alert fired at 14:09 UTC, three minutes after the first customer-visible failures. The on-call engineer began investigating at 14:13 UTC.

The publishing success-rate alert did not fire until 14:21 UTC because it evaluated a 15-minute rolling window. This delayed confirmation that the issue affected the entire publishing pipeline rather than a subset of workers.

## Timeline

All times are in UTC.

- **13:52** — Version `publisher-api-2026.07.08.3` begins deployment.
- **14:01** — Deployment completes across all API instances.
- **14:06** — Database query latency begins increasing as scheduled publishing traffic rises.
- **14:09** — Database connection-pool utilization exceeds 95%; alert fires.
- **14:13** — On-call engineer acknowledges the alert and begins investigation.
- **14:17** — Publishing workers begin timing out while claiming jobs.
- **14:21** — Publishing success-rate alert fires.
- **14:24** — Incident declared SEV-1; incident commander and database engineer paged.
- **14:31** — Team identifies a sharp increase in execution time for the job-status query introduced in the latest release.
- **14:38** — Deployment rollback begins.
- **14:46** — Rollback completes, but database load remains elevated because queued retries continue.
- **14:52** — Automatic publishing retries are disabled.
- **15:03** — Database connection utilization begins decreasing.
- **15:11** — Publishing workers are restarted at 25% capacity.
- **15:25** — New publishing requests begin completing successfully.
- **15:41** — Worker capacity increased to 60%; backlog begins shrinking.
- **16:07** — Worker capacity restored to 100%.
- **16:53** — Publishing backlog returns to normal levels; incident resolved.
- **18:07** — All episodes delayed during the incident have been processed or placed in a recoverable customer-visible state.

## Root Cause

The deployment added a query used to determine whether an episode already had an active publishing job:

```sql
SELECT id
FROM publishing_jobs
WHERE podcast_id = $1
  AND episode_id = $2
  AND state NOT IN ('completed', 'cancelled')
ORDER BY created_at DESC
LIMIT 1;
```

The production `publishing_jobs` table contained approximately 410 million rows. Although indexes existed on `podcast_id` and `episode_id` separately, there was no composite index covering the query’s filters and sort order.

In staging, the table contained fewer than two million rows, so the query completed quickly and did not trigger performance-test thresholds. In production, the database frequently scanned and sorted thousands of rows per request.

The query was executed twice for each publishing attempt. During the afternoon scheduled-publishing peak, it consumed database CPU and held connections long enough to exhaust the API and worker connection pools.

Publishing workers interpreted connection timeouts as transient failures and retried jobs using exponential backoff. Because thousands of jobs failed within a short period, their retry windows overlapped. This retry amplification kept database utilization high after the application rollback and prevented immediate recovery.

## Contributing Factors

- The staging database did not represent production table size or data distribution.
- Query review relied on staging execution time rather than a production-scale query plan.
- The publishing API and workers shared the same database cluster and connection budget.
- Retry logic did not include a global circuit breaker or concurrency limit.
- The publishing success-rate alert used a 15-minute rolling window.
- The deployment process did not automatically compare database query latency against the previous release.
- Customers could retry apparently failed uploads, producing duplicate drafts and additional publishing jobs.

## Resolution and Recovery

We rolled back the release containing the query. Because queued retries continued to saturate the database, rollback alone did not restore service.

We then disabled automatic retries, allowed existing database work to drain, and restarted publishing workers at reduced capacity. Capacity was increased gradually while monitoring database CPU, connection usage, job latency, and error rate.

After the database stabilized, we re-enabled retries with a temporary concurrency cap. A reconciliation job identified failed and stalled publishing jobs and safely requeued them. Duplicate drafts created during the incident were flagged for cleanup without deleting customer content automatically.

## What Went Well

- Existing RSS feeds and audio delivery were isolated from the publishing path and remained available.
- Database connection-pool monitoring detected the issue within three minutes.
- The deployment was reversible without a schema rollback.
- Job processing was idempotent, allowing stalled jobs to be replayed without duplicating published episodes.
- Support and status-page communications began within 31 minutes of the first customer impact.

## What Went Poorly

- The publishing-specific alert confirmed the outage too slowly.
- The rollback did not stop retries, which extended recovery time.
- Staging performance results created false confidence in the query.
- Shared database capacity allowed publishing failures to delay analytics dashboards.
- The customer interface encouraged manual retries without explaining that processing might still complete.
- The incident runbook did not document how to pause retries globally.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add a composite index on `publishing_jobs(podcast_id, episode_id, state, created_at DESC)` and validate its production query plan | Database Team | P0 | July 17, 2026 |
| Add a global retry circuit breaker and per-service concurrency limits | Publishing Team | P0 | July 24, 2026 |
| Reduce publishing success-rate alerting window from 15 minutes to 5 minutes | Reliability Team | P0 | July 17, 2026 |
| Create a production-scale anonymized dataset for database performance testing | Developer Infrastructure | P1 | August 14, 2026 |
| Require query-plan review for changes accessing tables above 100 million rows | Database Team | P1 | July 31, 2026 |
| Separate publishing-worker and API database connection budgets | Platform Team | P1 | August 7, 2026 |
| Add deployment monitoring for per-query latency regressions | Observability Team | P1 | August 21, 2026 |
| Add a documented and tested procedure for pausing publishing retries | Reliability Team | P1 | July 24, 2026 |
| Update the publishing UI to distinguish delayed jobs from terminal failures | Web Application Team | P2 | August 28, 2026 |
| Add duplicate-draft detection for repeated uploads of the same audio and metadata | Publishing Team | P2 | September 11, 2026 |
| Evaluate isolating analytics and publishing workloads onto separate database clusters | Architecture Group | P2 | September 18, 2026 |

## Lessons Learned

A reversible application deployment does not guarantee a fast recovery when the failed version creates persistent asynchronous work. Rollback procedures must account for queued jobs, retries, and other load that survives the deployment itself.

We also need performance tests that reflect production data size and distribution. Query latency against a small staging dataset was not a meaningful predictor for a table containing hundreds of millions of rows. Future changes to high-volume tables will require production-scale query-plan validation before release.