# Postmortem: RSS Feed API Outage and Delayed Episode Delivery

## Summary

On 2026-06-18, the podcast hosting platform experienced a partial outage affecting RSS feed generation, episode publishing, and analytics ingestion. For 1 hour and 42 minutes, many podcast RSS feeds returned elevated `500` errors or stale cached responses. Newly published episodes were delayed from appearing in subscriber apps, and analytics events were queued but not lost.

The incident was caused by a database migration that added a new nullable column to the `episodes` table, followed by an application deploy that assumed the column was always present in read replicas. One read replica lagged behind schema replication, causing feed-rendering workers connected to that replica to fail when querying episode metadata.

## Impact

- Incident window: 2026-06-18 14:07-15:49 UTC
- Severity: SEV-2
- Affected customers: Approximately 18% of hosted podcasts
- Failed RSS requests: ~1.3 million
- Delayed episode availability: Up to 96 minutes
- Analytics ingestion: Delayed, no confirmed data loss
- Admin dashboard: Episode publish status was intermittently incorrect
- Payments, account login, and media file downloads were not affected

Listeners using podcast apps that refreshed during the incident may have seen missing episodes, stale feeds, or temporary feed fetch failures.

## Timeline

All times are UTC.

| Time | Event |
| --- | --- |
| 14:02 | Database migration adding `episodes.transcript_status` begins. |
| 14:06 | Application deploy starts rolling out feed-rendering code that reads `transcript_status`. |
| 14:07 | First spike in RSS feed `500` responses detected. |
| 14:10 | On-call engineer receives alert for elevated feed API error rate. |
| 14:14 | Initial investigation focuses on CDN and cache invalidation due to high RSS request volume. |
| 14:21 | Customer support reports multiple tickets about missing newly published episodes. |
| 14:29 | Engineering identifies errors from feed-rendering workers: missing column `transcript_status`. |
| 14:34 | Team confirms one read replica has not completed schema replication. |
| 14:38 | Feed workers are temporarily routed away from the lagging read replica. Error rate begins dropping. |
| 14:47 | Application deploy is rolled back to avoid further reads from the new column. |
| 15:05 | RSS feed error rate returns to baseline. |
| 15:18 | Backlogged publish jobs begin processing normally. |
| 15:34 | Analytics ingestion queue drains to normal levels. |
| 15:49 | Incident declared resolved after validation across feed API, publish pipeline, and support reports. |

## Root Cause

The outage was caused by an unsafe deploy sequence involving a database schema change and application code change.

The migration added a new `transcript_status` column to the primary database. The following application deploy introduced feed-rendering logic that selected this column for every episode. The code assumed all database replicas had the new schema.

One read replica was experiencing elevated replication lag and had not yet applied the schema change. Feed-rendering workers using that replica failed with SQL errors when querying the missing column. Because RSS feed generation is on the critical path for podcast distribution, these failures surfaced directly as RSS `500` responses.

## Contributing Factors

- The deploy checklist did not require verification that schema changes had reached all read replicas.
- Feed-rendering workers used read replicas without schema-version awareness.
- The application query selected the new column unconditionally instead of using a backward-compatible rollout.
- Alerts detected feed API failures quickly, but the initial runbook emphasized CDN issues and delayed database investigation.
- The publish pipeline retried failed feed refresh jobs aggressively, increasing load on the affected workers.

## What Went Well

- The on-call alert fired within three minutes of the first elevated error rate.
- Routing feed workers away from the lagging replica quickly reduced customer-facing errors.
- Analytics events were queued durably and recovered without confirmed loss.
- Support escalated customer reports quickly, helping confirm user-visible impact.

## What Could Be Improved

- Schema rollout safety needs to account for replica lag.
- Feed generation should degrade gracefully when optional metadata is unavailable.
- Runbooks should include database schema mismatch checks for feed API errors.
- Publish job retries need better backoff during platform-wide failures.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add automated replica schema verification before application deploys that depend on migrations. | Database Platform | 2026-07-02 | Open |
| Update deploy process to require backward-compatible application changes for all schema migrations. | Core Platform | 2026-07-05 | Open |
| Add schema-version checks to feed-rendering workers before selecting newly added columns. | Feed Engineering | 2026-07-10 | Open |
| Change optional feed metadata reads to fail closed and omit unavailable fields instead of failing the whole feed. | Feed Engineering | 2026-07-12 | Open |
| Update RSS outage runbook with database replica and migration diagnostics. | SRE | 2026-06-28 | Open |
| Tune publish job retry policy with exponential backoff and circuit breaking. | Publishing Team | 2026-07-08 | Open |
| Add dashboard panel for replica schema lag and migration status. | Observability | 2026-07-15 | Open |

## Prevention

Future schema changes will follow an expand-and-contract pattern:

1. Deploy additive database changes.
2. Verify schema availability across all replicas.
3. Deploy application code that can tolerate both old and new schemas.
4. Backfill data if needed.
5. Enforce new assumptions only after verification.

This incident was preventable. The primary fix is not just better detection, but a safer deploy model for database-backed feed generation.