# Postmortem: Podcast Delivery and Publishing Outage **Incident date:** July 24, 2026 **Duration:** 1 hour 47 minutes **Severity:** SEV-1 **Status:** Resolved ## Summary On July 24, our podcast hosting platform experienced a partial outage affecting episode publishing, RSS feed delivery, and audio playback. A database migration introduced an inefficient query path in the feed-generation service. As traffic increased, the query exhausted the primary database connection pool and caused requests to queue across dependent services. Existing episodes remained available through some cached CDN locations, but uncached audio requests and RSS feed refreshes frequently failed. We restored service by disabling the new query path, increasing database capacity temporarily, and draining the accumulated request backlog. ## Customer Impact From 14:12 to 15:59 UTC: - 38% of RSS feed requests returned HTTP 500 or 503 responses. - 24% of audio download requests failed on the first attempt. - 6,842 episode publishing jobs were delayed. - New episodes took up to 96 minutes to appear in podcast directories. - Analytics ingestion continued, but dashboard data was delayed by approximately three hours. - No audio files, episode metadata, subscriber data, or analytics events were lost. Listeners using cached feeds or cached audio were generally unaffected. Customers publishing during the incident experienced the most significant impact. ## Timeline All times are in UTC. - **13:45** — Version `feed-service-2026.07.24.3` was deployed to 10% of production instances. - **13:55** — Automated checks passed; deployment proceeded to 100%. - **14:07** — Database query latency began increasing as directory polling traffic reached its normal afternoon peak. - **14:12** — RSS feed error rate exceeded the alert threshold. - **14:14** — The on-call engineer acknowledged the alert and began investigating. - **14:19** — Audio delivery errors increased as shared database connections became unavailable to the media authorization service. - **14:24** — The incident was declared SEV-1 and an incident commander was assigned. - **14:31** — Engineers identified database connection-pool exhaustion but initially attributed it to an increase in publishing traffic. - **14:38** — Database capacity was increased by 50%. Error rates briefly improved, then rose again as queued requests resumed. - **14:46** — Query tracing linked the load to the new feed entitlement lookup introduced in the latest deployment. - **14:52** — The team halted publishing workers to reduce database contention. - **15:01** — Rollback began. The rollback did not immediately complete because several instances were stuck terminating active requests. - **15:14** — Traffic was shifted away from affected instances, and the previous feed-service version reached 80% capacity. - **15:22** — RSS feed success rates returned above 99%. - **15:30** — Publishing workers were restarted at reduced concurrency. - **15:41** — Audio delivery error rates returned to baseline. - **15:59** — The publishing backlog was draining normally, and the incident was marked resolved. - **17:38** — All delayed publishing jobs had completed. - **18:54** — Analytics dashboards caught up with real-time ingestion. ## Root Cause The outage was caused by a change to the feed-generation service intended to support per-episode subscriber entitlements. The new code loaded entitlement records while generating each RSS feed. The underlying query was expected to use a composite index on `podcast_id`, `episode_id`, and `status`. That index existed in staging but had not been applied to the largest production database shard because an earlier migration had timed out and been marked as non-blocking. The application deployment did not verify that the required index was present on every shard. Without the index, the query performed repeated scans of the entitlement table. Feed generation executes at high volume because podcast directories poll feeds frequently, so the additional database work quickly consumed available connections. Retries from clients and internal services amplified the load. Services unrelated to feed generation were affected because they shared the same database cluster and connection proxy. ## Contributing Factors - The deployment check verified application health but did not measure query performance under representative feed-polling traffic. - Schema migration status was tracked globally rather than per database shard. - The feed service retried database timeouts twice without exponential backoff or jitter. - Feed generation and media authorization shared the same database connection pool. - The rollback process waited too long for requests executing slow database queries to terminate. - The initial alert identified elevated RSS errors but did not directly identify database connection saturation. - Staging contained less entitlement data and had the expected index, so performance testing did not reproduce the failure. ## Detection The incident was detected by an automated alert on the RSS feed HTTP error rate. Detection occurred five minutes after database latency began rising and two minutes after customer-visible failures crossed the incident threshold. Database connection-pool saturation was visible in internal dashboards, but no paging alert existed for sustained pool wait time. This delayed identification of the shared dependency failure. ## Resolution and Recovery We restored service by: 1. Pausing publishing workers to reduce database demand. 2. Shifting feed traffic to instances running the previous application version. 3. Temporarily increasing database capacity. 4. Disabling retries for the affected query path. 5. Restarting publishing workers with reduced concurrency. 6. Draining delayed publishing and analytics workloads after interactive traffic stabilized. The missing index was added later through an online migration after confirming it would not introduce additional production load. ## What Went Well - The on-call engineer acknowledged the initial alert within two minutes. - Query tracing identified the problematic application change within 40 minutes. - Audio files and analytics events remained durable throughout the incident. - Traffic shifting allowed recovery without waiting for slow instances to terminate cleanly. - Customer support and status-page updates began within 25 minutes of the first alert. ## What Went Poorly - The application was deployed without verifying its required schema state on every production shard. - Retry behavior increased load during database saturation. - Shared database resources allowed a feed-generation failure to affect audio authorization and publishing. - The first capacity increase treated the symptom and briefly intensified the backlog. - The status page initially described the issue as publishing-only, understating listener impact. - Publishing customers did not receive clear guidance about whether retrying submissions was safe. ## Action Items | Action | Owner | Priority | Due Date | |---|---|---:|---:| | Add deployment gates that verify required indexes and migrations on every production shard | Database Platform | P0 | August 7, 2026 | | Add a circuit breaker around entitlement lookups and serve the last valid feed when the dependency is degraded | Feed Team | P0 | August 14, 2026 | | Separate feed-generation and media-authorization database connection pools | Platform Infrastructure | P0 | August 21, 2026 | | Add exponential backoff, jitter, and retry budgets to database clients | Developer Platform | P1 | August 21, 2026 | | Page on sustained connection-pool wait time and database proxy saturation | Observability | P1 | August 7, 2026 | | Add production-scale feed-polling traffic to pre-deployment load tests | Reliability Engineering | P1 | August 28, 2026 | | Track migration completion and failures per shard in the deployment dashboard | Database Platform | P1 | August 14, 2026 | | Add a forced traffic-drain option to the feed-service rollback procedure | Runtime Platform | P1 | August 14, 2026 | | Update incident templates to distinguish publishing, feed, playback, and analytics impact | Incident Management | P2 | August 7, 2026 | | Document customer retry guidance for delayed or failed episode publishing | Customer Support Engineering | P2 | August 7, 2026 | ## Lessons Learned Changes that depend on database schema must verify the actual production schema rather than trusting migration history. High-volume read paths also need degradation strategies that preserve cached customer content when a personalization dependency fails. We are treating database isolation, migration verification, and bounded retries as required safeguards for all critical delivery services.