# Postmortem: Podcast Episode Publishing and Feed Delivery Outage

**Date:** 2026-06-18  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** Episode publishing, RSS feed generation, analytics ingestion delay

## Summary

On June 18, 2026, a deployment to the feed generation service caused newly published podcast episodes to fail to appear in customer RSS feeds. Existing feeds continued to serve cached content, but new episodes, metadata edits, and scheduled releases were not reflected for many customers.

The issue began at 09:14 UTC and was fully resolved by 10:56 UTC after the deployment was rolled back and feed generation jobs were replayed.

## Customer Impact

- Approximately 31% of active shows were unable to publish new episodes to RSS during the incident.
- Scheduled releases between 09:15 UTC and 10:45 UTC were delayed.
- Some podcast directories and listening apps did not receive updated feeds until after recovery.
- Analytics for affected episodes were delayed by up to 2 hours, but no analytics data was lost.
- No uploaded media files were lost or corrupted.

## Timeline

All times are UTC.

- **09:02** - Feed generation service deployment begins.
- **09:14** - First elevated error rate detected in `feed-render-worker`.
- **09:18** - On-call engineer receives alert for failed feed generation jobs.
- **09:23** - Support reports customer tickets about scheduled episodes not appearing.
- **09:29** - Incident declared as SEV-1.
- **09:34** - Initial investigation identifies failures in feed rendering for shows with dynamic ad markers.
- **09:41** - Deployment is suspected as the cause.
- **09:47** - Rollback initiated.
- **10:03** - Rollback completes, but backlog remains high.
- **10:12** - Feed generation jobs are replayed from the failed job queue.
- **10:31** - Error rate returns to baseline.
- **10:44** - Delayed scheduled episodes confirmed visible in generated RSS feeds.
- **10:56** - Incident resolved.
- **11:30** - Customer-facing status page updated with final resolution note.

## Root Cause

The outage was caused by a schema mismatch introduced in the feed generation service.

A new version of the service expected dynamic ad marker metadata to always include a `placement_id`. Older records in the database did not consistently include this field. When the service encountered one of these records, it raised an unhandled exception and failed the entire feed generation job.

The deployment passed automated tests because test fixtures only covered newly created ad marker records and did not include legacy records with missing fields.

## Contributing Factors

- Feed generation jobs failed at the show level instead of skipping invalid optional ad marker metadata.
- Backward compatibility checks were not included in the deployment validation process.
- The alert identified job failures but did not immediately show that customer-facing RSS feeds were stale.
- The failed job queue replay process was manual, which slowed recovery.

## Detection

The issue was first detected by an automated alert on elevated feed generation job failures. Customer support reports confirmed that the failure was customer-visible.

## Resolution

The feed generation service was rolled back to the previous stable version. After rollback, failed feed generation jobs were replayed, and the backlog was monitored until all affected feeds were regenerated successfully.

## What Went Well

- Automated alerts fired within minutes of the first failures.
- The rollback path was available and completed successfully.
- No media uploads, show records, or analytics events were permanently lost.
- Support and engineering coordination was effective once the incident was declared.

## What Went Wrong

- The deployment was not validated against legacy ad marker records.
- A single malformed optional metadata field caused full feed generation failure.
- Recovery required manual replay of failed jobs.
- The customer-facing status page was updated later than expected.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add backward compatibility test fixtures for legacy ad marker records | Feed Platform | P0 | 2026-06-21 |
| Make feed rendering tolerant of missing optional ad metadata | Feed Platform | P0 | 2026-06-24 |
| Add stale RSS feed freshness alerting | Observability | P1 | 2026-06-28 |
| Automate replay of failed feed generation jobs after rollback | Infrastructure | P1 | 2026-07-05 |
| Update deployment checklist to include legacy data validation | Engineering Productivity | P1 | 2026-07-05 |
| Improve incident status page ownership and timing expectations | Support Ops | P2 | 2026-07-12 |

## Prevention

Future feed generation changes will be tested against representative legacy customer data before deployment. Optional metadata fields will be treated defensively, and feed generation failures will be isolated so one invalid record cannot prevent an entire show feed from updating.