# Postmortem: Podcast Episode Delivery Outage

**Incident Date:** 2026-07-08  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services Affected:** RSS feed generation, episode downloads, analytics ingestion

## Summary

On July 8, 2026, a database migration caused elevated latency and partial failures across the podcast hosting platform. During the incident, many podcast RSS feeds failed to load, some episode downloads returned errors, and analytics events were delayed.

The outage was caused by a missing index on a newly introduced query path in the feed rendering service. Under production traffic, the query triggered full table scans against the `episodes` table, saturating the primary database CPU and causing connection pool exhaustion across dependent services.

## Customer Impact

- Approximately 38% of RSS feed requests returned `500` or timed out.
- Episode downloads were intermittently unavailable for approximately 21% of listeners.
- Publisher dashboards loaded slowly or failed during the incident window.
- Analytics ingestion was delayed by up to 3 hours, but no analytics data was lost.
- 14 enterprise customers opened support tickets during the incident.

## Timeline

All times are in UTC.

| Time | Event |
|---|---|
| 14:03 | Migration adding scheduled episode visibility rules was deployed. |
| 14:11 | Database CPU increased from 42% to 91%. |
| 14:16 | Feed rendering latency exceeded the alert threshold. |
| 14:19 | On-call engineer acknowledged the alert. |
| 14:24 | Support reported increased customer complaints about unavailable feeds. |
| 14:31 | Incident declared as SEV-1. |
| 14:38 | Engineers identified database saturation as the immediate failure mode. |
| 14:47 | Feed rendering workers were scaled down to reduce database pressure. |
| 15:02 | Slow query logs identified repeated scans on the `episodes` table. |
| 15:14 | Missing composite index was confirmed as the root cause. |
| 15:21 | Hotfix deployed to bypass the new visibility query for public RSS feeds. |
| 15:33 | Database CPU returned below 60%. |
| 15:45 | RSS feed error rate returned to normal levels. |
| 15:56 | Incident resolved. |
| 17:02 | Analytics backlog fully processed. |

## Root Cause

A recent feature added support for scheduled episode visibility by checking episode publish windows during RSS feed generation. The new query filtered by `podcast_id`, `visibility_status`, and `publish_at`.

The migration included the schema change but did not include the required composite index:

```sql
(podcast_id, visibility_status, publish_at)
```

In staging, the issue was not detected because the test dataset was too small and did not match production episode distribution. In production, high-volume podcasts with large episode catalogs caused the database to perform full table scans for frequently requested RSS feeds.

As database CPU became saturated, feed rendering requests held connections longer. This exhausted connection pools in the feed service, which then caused cascading failures in episode download authorization and publisher dashboard APIs.

## Resolution

The immediate mitigation was to deploy a hotfix that restored the previous feed query behavior for public RSS feeds. After traffic stabilized, the missing index was created using an online migration. Analytics workers were then allowed to drain the backlog.

## What Went Well

- Alerting detected the elevated feed latency within minutes.
- Slow query logs made the problematic query identifiable.
- The hotfix was small, reversible, and deployed quickly.
- Analytics events were queued durably, so no data was lost.

## What Went Poorly

- The migration review did not require query plan validation.
- Staging data was not representative of production scale.
- Feed service connection pool exhaustion affected adjacent services.
- Customer support learned about the issue from customers before the incident channel was created.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---:|---|
| Add required composite index for scheduled visibility queries | Database Team | 2026-07-09 | Complete |
| Require `EXPLAIN ANALYZE` output for migrations that add production query paths | Platform Team | 2026-07-15 | Open |
| Add production-scale dataset sampling to staging load tests | Infrastructure Team | 2026-07-22 | Open |
| Add circuit breaker between feed rendering and download authorization | Media Delivery Team | 2026-07-26 | Open |
| Reduce feed service database connection pool limits to prevent saturation cascades | Platform Team | 2026-07-12 | Open |
| Create automated support notification when SEV-1 incidents are declared | Support Engineering | 2026-07-18 | Open |
| Add dashboard panel for top slow feed-rendering queries by podcast ID | Observability Team | 2026-07-19 | Open |

## Preventative Measures

Future schema and query changes that affect feed generation must include query plan review, load testing against representative data, and explicit rollback steps. The feed rendering service will also be isolated further so database saturation in one read path cannot degrade episode download authorization.