# Postmortem: Delayed Podcast Publishing and Feed Availability

**Incident date:** July 14, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 14, our publishing pipeline experienced a database connection surge that delayed new episode publication and caused intermittent errors when listeners and podcast directories requested RSS feeds.

The incident began after a routine deployment introduced an unbounded retry loop in the service responsible for generating RSS feeds. A brief increase in database latency caused feed-generation jobs to fail and retry immediately. These retries exhausted the primary database connection pool, affecting both background publishing jobs and synchronous feed requests.

We restored service by disabling the new retry behavior, draining the job backlog, and temporarily increasing database capacity. All delayed episodes were published successfully. No podcast metadata or media files were lost.

## Customer Impact

Between 09:18 and 12:05 UTC:

- Approximately 31% of newly scheduled episodes were published late.
- The median publication delay was 42 minutes; the longest delay was 2 hours 21 minutes.
- Approximately 8% of RSS feed requests returned HTTP 500 or 503 responses.
- Apple Podcasts, Spotify, and other podcast directories experienced delayed feed refreshes for affected shows.
- The creator dashboard intermittently displayed episodes as “Processing” after uploads had completed.
- Existing audio files remained available through previously cached feed data and direct media URLs.
- No customer data was corrupted or permanently lost.

A total of 12,840 shows and approximately 4.2 million feed requests were affected.

## Detection

Our availability alert fired at 09:25 UTC after the RSS feed error rate exceeded 5% for five minutes. A separate alert for publishing queue depth fired two minutes later.

The initial database connection saturation was visible at 09:20 UTC, but the corresponding alert threshold required 15 minutes of sustained saturation and did not trigger before the customer-facing alerts.

## Timeline

All times are in UTC on July 14, 2026.

- **08:52** — Version `feeds-api-2026.07.14.1` begins rolling out to production.
- **09:07** — Deployment completes successfully. Initial health checks and error rates are normal.
- **09:16** — A scheduled analytics workload increases read latency on the primary metadata database.
- **09:18** — Feed-generation jobs begin timing out and retrying without delay.
- **09:20** — Database connections reach 95% utilization. Publishing throughput starts declining.
- **09:23** — RSS feed error rates exceed the normal baseline.
- **09:25** — RSS feed availability alert fires. The on-call engineer begins investigation.
- **09:27** — Publishing queue-depth alert fires.
- **09:34** — Incident declared SEV-1. Database and publishing teams are paged.
- **09:42** — Responders identify high connection churn from feed-generation workers.
- **09:49** — Worker concurrency is reduced from 400 to 100 as an initial mitigation.
- **09:55** — Connection utilization declines briefly, then rises again as queued jobs continue retrying.
- **10:08** — The team correlates the retry traffic with the morning deployment.
- **10:19** — A feature flag disables the new retry implementation for synchronous feed requests.
- **10:27** — RSS feed error rates begin to fall, but publishing remains delayed.
- **10:41** — Feed-generation workers are paused to allow interactive requests and core publishing transactions to recover.
- **10:53** — Database connection utilization returns below 70%.
- **11:02** — A patched worker version with exponential backoff and retry limits begins deployment.
- **11:19** — Patched workers resume processing at reduced concurrency.
- **11:31** — Publishing throughput exceeds incoming job volume; the backlog begins shrinking.
- **11:47** — RSS feed availability returns to normal.
- **12:05** — All overdue publishing jobs complete. Incident marked resolved.
- **14:30** — Validation confirms that no episode or feed data was lost.

## Root Cause

The direct cause was an unbounded, immediate retry loop introduced in the RSS feed-generation service.

The deployment changed retry handling from a shared library with exponential backoff to service-local logic intended to distinguish transient database failures from permanent feed-validation errors. The new implementation correctly classified database timeouts as retryable but did not enforce a maximum attempt count or delay between attempts.

When database latency increased at 09:16 UTC, hundreds of feed-generation jobs timed out simultaneously. Each job immediately opened a new transaction and retried. This created a retry storm that exhausted the primary database connection pool.

The publishing pipeline and synchronous RSS feed API shared the same database pool and primary database cluster. As a result, background retries prevented customer-facing requests and publishing transactions from acquiring connections.

## Contributing Factors

- The retry implementation had unit tests for error classification but no tests for backoff, retry limits, or sustained failure behavior.
- Load testing covered normal publishing volume but did not simulate database timeouts.
- Background workers and customer-facing feed requests shared database capacity without reserved connection limits.
- Worker concurrency was configured globally, making it difficult to reduce concurrency for only the failing job type.
- The deployment passed health checks because the issue required a downstream latency increase to become visible.
- The database saturation alert was configured with a longer evaluation window than the customer-facing availability alerts.
- The analytics workload ran against the primary database because a read-replica routing rule did not cover one recently added query path.

## Resolution and Recovery

We mitigated the incident by:

1. Reducing worker concurrency.
2. Disabling the newly deployed retry behavior.
3. Pausing feed-generation workers long enough for interactive traffic to recover.
4. Deploying a patch that limited retries to three attempts with exponential backoff and jitter.
5. Temporarily increasing database capacity while the publishing backlog drained.
6. Reprocessing delayed publishing jobs and validating feed state against stored episode metadata.

## What Went Well

- Customer-facing availability alerts detected the issue within seven minutes of the first failed requests.
- Feature flags allowed us to disable part of the new behavior without a full rollback.
- The publishing queue retained all jobs, so no episodes required manual re-uploading.
- Cross-functional incident response began within 20 minutes of the initial alert.
- Existing media delivery remained healthy and isolated from the metadata database failure.

## What Went Poorly

- The initial concurrency reduction provided only temporary relief and delayed identification of the retry loop.
- Our dashboards did not clearly separate original database requests from retries.
- The shared connection pool allowed background work to starve customer-facing traffic.
- Status-page communication began 34 minutes after customer impact started.
- We lacked an automated mechanism to pause or rate-limit a single job type.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add a shared retry policy with mandatory attempt limits, exponential backoff, and jitter | Platform Runtime | P0 | July 21, 2026 |
| Add integration tests that simulate sustained database timeouts | Feed Platform | P0 | July 24, 2026 |
| Enforce separate database connection budgets for interactive requests and background workers | Database Engineering | P0 | July 31, 2026 |
| Add per-job-type concurrency and pause controls | Publishing Infrastructure | P1 | August 7, 2026 |
| Route the affected analytics query path to a read replica | Data Platform | P1 | July 22, 2026 |
| Alert on retry rate and connection-acquisition latency | Observability | P1 | July 24, 2026 |
| Reduce the database saturation alert evaluation window from 15 minutes to 5 minutes | Site Reliability Engineering | P1 | July 20, 2026 |
| Add database degradation scenarios to publishing load tests | Performance Engineering | P1 | August 14, 2026 |
| Update deployment review requirements for retry and timeout changes | Engineering Productivity | P2 | August 14, 2026 |
| Automate status-page updates when a SEV-1 incident is declared | Incident Management | P2 | August 21, 2026 |

## Lessons Learned

Retry behavior must be treated as a capacity-management feature, not only as error handling. A retry can amplify a downstream failure, particularly when many workers fail at the same time.

We also need stronger isolation between background processing and customer-facing traffic. The publishing queue protected data durability, but shared database resources allowed recoverable background failures to become a platform-wide availability incident.