# Postmortem: Podcast Publishing and RSS Feed Outage

## Summary

On 2026-06-18, the podcast hosting platform experienced a partial outage affecting episode publishing, RSS feed updates, and audio processing. Newly uploaded episodes were accepted by the API but were not reliably processed or published to customer RSS feeds.

The incident lasted 1 hour and 47 minutes. Existing podcast feeds and previously published episodes remained available.

## Impact

- 18,420 episode uploads were delayed.
- 6,870 RSS feed updates were not published on time.
- 1,140 customers saw episodes stuck in `processing` or `scheduled` states.
- Podcast listeners were not able to access newly published episodes through podcast apps until feeds caught up.
- No audio files or metadata were lost.
- No customer billing or account data was affected.

## Timeline

All times are UTC.

| Time | Event |
| --- | --- |
| 09:12 | A database migration for the episode processing queue was deployed. |
| 09:18 | Queue latency began increasing for the `publish-feed` worker group. |
| 09:24 | First customer support ticket reported scheduled episodes not appearing in Apple Podcasts or Spotify. |
| 09:29 | On-call engineer received an alert for elevated queue age. |
| 09:36 | Engineering confirmed that episode uploads were succeeding but downstream publishing jobs were stalled. |
| 09:44 | The migration was identified as a likely contributor due to increased lock contention on the `episode_jobs` table. |
| 09:51 | Worker autoscaling was paused to reduce database pressure. |
| 10:03 | A rollback of the migration was started. |
| 10:17 | Database lock contention returned to normal levels. |
| 10:26 | Publishing workers resumed processing jobs at normal throughput. |
| 10:41 | RSS feed publishing backlog was fully drained. |
| 10:59 | Audio transcoding backlog was fully drained. |
| 11:07 | Incident was marked resolved. |

## Root Cause

The outage was caused by a database migration that added a new nullable column and index to the `episode_jobs` table. The index creation was intended to run concurrently, but the migration framework generated a non-concurrent index statement because the migration was wrapped in a transaction.

This caused lock contention on a high-write table used by episode processing and RSS feed publishing workers. As worker retries increased, database write pressure grew further, causing queue latency to spike.

The platform continued accepting uploads, but publishing jobs could not consistently update job state or enqueue RSS feed regeneration tasks.

## Contributing Factors

- The migration was deployed during a high-traffic publishing window.
- The migration review checklist did not explicitly require verification of concurrent index creation.
- Worker retry behavior used aggressive backoff settings that amplified database load.
- Queue latency alerts fired only after the backlog was already customer-visible.
- The admin dashboard showed upload success but did not clearly distinguish upload completion from feed publication completion.

## Resolution

The migration was rolled back, worker autoscaling was paused, and the publishing backlog was allowed to drain under controlled concurrency. Once database lock contention returned to normal, worker concurrency was gradually restored.

No manual customer data repair was required.

## Customer Communication

Support notified affected customers that publishing was delayed and that uploads did not need to be retried. A status page update was posted at 09:42 UTC and marked resolved at 11:12 UTC.

## What Went Well

- Uploads were durably stored and no customer media was lost.
- Existing RSS feeds and published audio remained available.
- The rollback restored service without requiring a full database restore.
- Support quickly identified a pattern from customer reports and escalated to engineering.

## What Did Not Go Well

- The migration reached production without confirming the generated SQL.
- Queue latency detection was slower than customer reports.
- Worker retries increased load on an already constrained database.
- Customer-facing publishing states were ambiguous during the outage.

## Action Items

| Action Item | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Add automated checks that block non-concurrent indexes on high-write tables. | Database Platform | High | 2026-06-25 |
| Update migration review checklist for lock behavior and generated SQL verification. | Engineering Productivity | High | 2026-06-27 |
| Move high-risk schema migrations outside peak publishing hours. | Release Engineering | Medium | 2026-07-01 |
| Reduce retry pressure from episode workers during database contention. | Media Platform | High | 2026-07-03 |
| Add alerts for publishing delay before customer-visible thresholds are reached. | Observability | High | 2026-07-05 |
| Improve dashboard states for upload complete, processing complete, and feed published. | Product Engineering | Medium | 2026-07-12 |
| Run a game day for queue backlog recovery and worker throttling. | Site Reliability | Medium | 2026-07-19 |

## Prevention

Future schema changes to queue and publishing tables will require generated SQL review, explicit lock analysis, and staged rollout approval. Worker retry policies will be adjusted so that database contention causes controlled degradation instead of retry amplification.