# Postmortem: Podcast Feed and Analytics Outage

**Date:** 2026-07-08  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 8, 2026, a database migration caused elevated latency and intermittent failures across RSS feed generation, episode publishing, and download analytics ingestion. Podcast listeners using major podcast apps experienced delayed or failed feed refreshes, while creators saw publishing delays and incomplete analytics during the incident window.

The outage was caused by an unbounded backfill query introduced during a schema migration for episode-level attribution metadata. The query saturated the primary database, causing connection pool exhaustion across multiple services.

## Customer Impact

- RSS feed requests returned `5xx` responses or timed out for approximately 18% of traffic.
- Episode publishing was delayed for 312 creators.
- Analytics events were queued but not processed in real time for 1 hour 42 minutes.
- No analytics data was permanently lost.
- Existing audio file delivery through the CDN was unaffected.
- Listener playback for already-downloaded or cached episodes was unaffected.

## Timeline

All times in UTC.

| Time | Event |
|---|---|
| 14:02 | Migration deployed as part of the scheduled `2026.07.08.3` release. |
| 14:07 | Database CPU exceeded 90%; feed generation latency began increasing. |
| 14:11 | First automated alert fired for elevated RSS feed `5xx` rate. |
| 14:15 | On-call engineer began investigation and identified database saturation. |
| 14:23 | Episode publishing workers began timing out due to connection pool exhaustion. |
| 14:31 | Incident declared SEV-1. |
| 14:38 | Release rollback initiated. |
| 14:46 | Application rollback completed, but database load remained elevated. |
| 14:52 | Migration backfill query identified as still running on the primary database. |
| 15:01 | Backfill query terminated manually. |
| 15:09 | Database CPU and connection usage began recovering. |
| 15:22 | RSS feed error rate returned to baseline. |
| 15:36 | Publishing queue drained. |
| 15:44 | Analytics ingestion backlog fully processed. |
| 15:48 | Incident resolved. |

## Root Cause

A database migration added a nullable `attribution_source` column to the `episodes` table and attempted to backfill historical rows in the same deploy. The backfill query updated approximately 94 million rows without batching or a limiting predicate.

This created long-running row locks and heavy write amplification on the primary database. As database CPU and I/O became saturated, application services relying on the primary database exhausted their connection pools. RSS feed generation, episode publishing, and analytics ingestion all experienced degraded availability as a result.

## Contributing Factors

- The migration review focused on schema compatibility but did not flag the backfill size.
- There was no automated guardrail preventing large unbatched updates in production migrations.
- Feed generation depended on live database reads instead of falling back to recently cached feed snapshots.
- The analytics ingestion service shared the same primary database connection pool class as customer-facing services.
- The migration was deployed during a high-traffic publishing window.

## What Went Well

- Alerts fired within five minutes of customer impact.
- Rollback was completed quickly once the issue was isolated.
- Analytics events were durably queued, preventing data loss.
- CDN audio delivery remained available throughout the incident.

## What Went Poorly

- The rollback did not stop the already-running database backfill.
- The migration lacked a documented execution plan for production data volume.
- Customer-facing status updates were delayed while engineering focused on diagnosis.
- Internal dashboards did not clearly separate database lock contention from general CPU saturation.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add CI checks to block unbatched production data migrations over configurable row thresholds. | Database Platform | P0 | 2026-07-15 |
| Require migration runbooks for any schema change involving backfills on tables over 10 million rows. | Engineering Managers | P0 | 2026-07-18 |
| Move historical episode attribution backfill to an idempotent batched background job. | Creator Platform | P1 | 2026-07-22 |
| Add database kill-switch documentation for long-running migration queries. | SRE | P1 | 2026-07-19 |
| Separate analytics ingestion database pools from customer-facing API pools. | Data Platform | P1 | 2026-07-26 |
| Implement cached RSS feed fallback when live generation exceeds latency thresholds. | Feed Infrastructure | P1 | 2026-08-02 |
| Update incident process to assign a communications lead within 10 minutes of SEV-1 declaration. | Incident Response | P2 | 2026-07-20 |

## Prevention

Future schema changes on high-volume tables must separate structural changes from data backfills. Backfills must be batched, rate-limited, observable, and independently stoppable. Customer-facing services should also degrade gracefully when database reads are impaired, especially for RSS feeds where cached content is usually preferable to failed responses.