# Postmortem: Podcast Feed and Episode Delivery Outage

**Date:** 2026-07-08  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** RSS feed generation, episode publishing, analytics ingestion, public episode pages

## Summary

On July 8, 2026, a database migration caused elevated lock contention on the primary metadata database. This prevented several core services from reading and writing podcast, episode, and feed metadata.

As a result, podcast RSS feeds intermittently returned stale data or 500 errors, newly published episodes were delayed, and public episode pages failed for a subset of shows. Audio file delivery was not directly affected, so already-downloaded episodes and direct media file playback continued to work.

The incident lasted 1 hour and 42 minutes from initial customer impact to full recovery.

## Customer Impact

- Approximately 38% of podcast RSS feed requests failed or returned stale responses during the peak of the incident.
- New episode publishing was delayed for 1,860 shows.
- 412 scheduled episodes missed their intended publish time.
- Public episode pages returned elevated 500 errors for approximately 24% of requests.
- Analytics events were buffered but delayed by up to 3 hours.
- Existing audio files remained available via CDN and object storage.
- No customer data was lost.

## Timeline

All times are in UTC.

| Time | Event |
| --- | --- |
| 14:02 | A schema migration began on the primary metadata database to add a new index used by the feed personalization service. |
| 14:07 | Database write latency increased above normal thresholds. No alert fired because the latency remained below the SEV-2 paging threshold. |
| 14:14 | RSS feed generation workers began timing out while reading episode metadata. |
| 14:18 | First customer support tickets reported missing newly published episodes. |
| 14:21 | Feed error rate exceeded 10%. On-call engineer was paged. |
| 14:25 | On-call confirmed elevated database lock waits and API timeout errors. Incident declared SEV-1. |
| 14:31 | Publishing queue was paused to reduce write pressure on the database. |
| 14:39 | Migration was identified as the likely source of lock contention. |
| 14:44 | Migration process was terminated. Lock contention began to drop but did not fully recover due to backed-up workers retrying aggressively. |
| 14:52 | Feed generation and publishing workers were rate-limited. |
| 15:03 | Database read latency returned to baseline. RSS feed error rate dropped below 2%. |
| 15:16 | Publishing queue was resumed at 25% capacity. |
| 15:31 | All delayed scheduled episodes had been published or re-queued successfully. |
| 15:44 | Feed error rate returned to normal levels. Incident marked resolved. |
| 17:02 | Analytics backlog fully drained. |

## Root Cause

The outage was caused by a database migration that attempted to create an index on a high-traffic table without using the database engine’s online index creation mode.

The migration acquired locks on the `episodes` table that blocked writes and slowed reads. Several services depend on this table for RSS feed generation, public episode pages, publishing workflows, and analytics enrichment.

The issue was amplified by retry behavior in feed generation and publishing workers. As database latency increased, workers retried failed operations too quickly, which added more load to an already constrained database.

## Contributing Factors

- The migration was reviewed for correctness but not for production locking behavior.
- The staging dataset was too small to reveal migration runtime or lock contention risk.
- Database lock wait alerts were configured as warning-level only and did not page.
- Worker retry logic used fixed short intervals instead of exponential backoff with jitter.
- Publishing and feed generation shared the same primary metadata database without sufficient isolation.

## Detection

The incident was first detected by customer support tickets and then confirmed by automated feed error-rate paging.

Earlier signals were available from database lock wait metrics, but those alerts were not configured to page the on-call engineer.

## Resolution

The migration was stopped, publishing workers were paused, and feed generation workers were rate-limited. Once database latency recovered, publishing was resumed gradually. Delayed scheduled episodes were reprocessed, and analytics backlogs drained without data loss.

## What Went Well

- On-call quickly identified the database as the shared failing dependency.
- Pausing publishing reduced write pressure and helped recovery.
- Analytics buffering worked as expected and prevented event loss.
- CDN delivery for existing audio files remained available.

## What Went Poorly

- The migration was not safe for a high-traffic production table.
- Alerting did not catch database lock contention early enough.
- Worker retries increased pressure during the incident.
- Customer support received reports before engineering had declared the incident.

## Action Items

| Action Item | Owner | Priority | Status |
| --- | --- | --- | --- |
| Require online index creation for all production migrations on high-traffic tables. | Database Platform | P0 | Open |
| Add automated migration checks for lock-prone operations in CI. | Developer Experience | P0 | Open |
| Promote database lock wait alerts to page when sustained for more than 3 minutes. | SRE | P0 | Open |
| Add exponential backoff with jitter to feed generation and publishing workers. | Backend Platform | P1 | Open |
| Create a production-scale staging dataset for migration testing. | Data Infrastructure | P1 | Open |
| Add a migration runbook with rollback and abort criteria. | Database Platform | P1 | Open |
| Separate publishing writes from feed read paths using read replicas or service-level isolation. | Architecture | P2 | Planned |
| Improve incident status automation so support and status page updates begin sooner. | Customer Operations | P2 | Open |

## Follow-Up

A corrected version of the index migration will be tested against a production-scale copy of the metadata database before being retried. The migration will use online index creation, reduced concurrency, and an explicit abort threshold for lock wait time.