# Postmortem: Podcast Episode Delivery Outage

**Date:** 2026-07-08  
**Duration:** 1 hour 42 minutes  
**Status:** Resolved  
**Severity:** SEV-1  
**Services affected:** RSS feed generation, media delivery API, listener analytics ingestion

## Summary

On July 8, 2026, a database migration caused elevated latency and partial unavailability across the podcast hosting platform. The incident primarily affected podcast RSS feed requests and episode audio URL resolution. During the outage, some listeners were unable to stream or download episodes, and some podcasters saw delayed analytics updates.

The issue was caused by an unbounded backfill query introduced during a schema migration for episode-level monetization metadata. The query saturated the primary PostgreSQL database, causing connection pool exhaustion in downstream services.

## Customer Impact

From **14:06 UTC to 15:48 UTC**:

- Approximately **38% of RSS feed requests** failed or timed out.
- Approximately **24% of episode playback requests** experienced errors or latency above 10 seconds.
- Podcast publishing workflows were degraded; some newly published episodes took up to 35 minutes to appear in feeds.
- Listener analytics ingestion was delayed by up to 2 hours.
- No customer data was lost.
- No incorrect episode files were served.

The most affected customers were high-traffic shows with large episode catalogs, especially those with feeds containing more than 500 episodes.

## Timeline

All times in UTC.

| Time | Event |
| --- | --- |
| 14:00 | Database migration for monetization metadata begins. |
| 14:06 | API latency alerts fire for RSS feed generation. |
| 14:09 | On-call engineer acknowledges the alert. |
| 14:12 | Customer support reports increased complaints about failed episode downloads. |
| 14:17 | Engineering identifies elevated PostgreSQL CPU and active connection count. |
| 14:23 | Feed generation workers begin timing out due to database connection pool exhaustion. |
| 14:29 | Incident declared SEV-1. |
| 14:34 | Migration process is paused, but database load remains elevated. |
| 14:41 | Engineering identifies long-running backfill query locking heavily accessed episode rows. |
| 14:46 | Backfill query is terminated manually. |
| 14:53 | Database CPU begins recovering, but API error rates remain elevated due to retry backlog. |
| 15:05 | Feed generation workers are restarted in batches to clear saturated pools. |
| 15:19 | Episode playback error rate returns below alert threshold. |
| 15:32 | RSS feed latency returns to normal operating range. |
| 15:48 | Incident resolved. |
| 17:21 | Delayed analytics ingestion fully catches up. |

## Root Cause

The outage was caused by a schema migration that included a backfill step updating monetization metadata across all historical episode records.

The migration query was not bounded by batch size and did not include rate limiting. On large podcast catalogs, it scanned and updated millions of rows on the primary database. This caused:

- High PostgreSQL CPU usage.
- Increased row lock contention on the `episodes` table.
- Slow queries in feed generation and media URL resolution paths.
- Database connection pool exhaustion across multiple application services.

The migration had been tested in staging, but staging did not contain production-scale episode history or representative large feeds. As a result, the performance impact was not detected before deployment.

## Contributing Factors

- The migration combined a schema change and a large data backfill in one deployment.
- Staging data volume was too small to expose the query behavior.
- Migration review focused on correctness, not production performance characteristics.
- Feed generation depended directly on primary database reads for hot paths.
- Application retries increased pressure on the database once latency rose.

## Detection

The incident was detected through automated latency and error-rate alerts for RSS feed generation. Customer support also reported an increase in inbound tickets related to playback failures shortly after the first alert.

Detection worked, but the initial alerts did not clearly identify the migration as the likely trigger. Engineers correlated the issue manually using deployment logs and database activity metrics.

## Resolution

Engineering paused the migration, terminated the active backfill query, and restarted affected worker pools in batches. Once database load dropped, RSS feed generation and playback APIs recovered. Analytics ingestion recovered later after queued events were processed.

## What Went Well

- Alerts fired within minutes of user-facing impact.
- The on-call engineer escalated quickly to database and platform engineers.
- No customer data was lost.
- Services recovered without requiring a full database failover.

## What Went Poorly

- The migration was allowed to run an unbounded production backfill.
- Staging did not accurately represent production data scale.
- The rollback plan did not account for a partially completed backfill.
- Customer-facing status updates lagged behind the engineering response.

## Action Items

| Action | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Split schema migrations and data backfills into separate deploy steps. | Platform | High | 2026-07-18 |
| Require batching and rate limits for all production backfills. | Database | High | 2026-07-18 |
| Add migration review checklist covering locks, indexes, query plans, and rollback behavior. | Engineering Enablement | High | 2026-07-22 |
| Create production-scale staging dataset for feed and episode tables. | Data Platform | Medium | 2026-08-02 |
| Add alert annotation for active migrations and deploys. | SRE | Medium | 2026-07-25 |
| Move RSS feed reads for published episode metadata to read replicas where safe. | Podcast Platform | Medium | 2026-08-09 |
| Improve customer status page update process for SEV-1 incidents. | Support Engineering | Medium | 2026-07-26 |

## Prevention

Future migrations that touch high-traffic tables must use bounded backfills, explicit query plans, and staged rollout controls. Large data updates will be executed separately from schema changes and monitored with automatic stop conditions for database CPU, lock wait time, and replication lag.