# Postmortem: Delayed Podcast Publishing and Feed Availability

**Incident date:** July 24, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 24, 2026, a deployment to our feed-generation service caused newly published episodes and metadata updates to remain queued instead of being processed. Existing podcast feeds continued to load from cache, but cache misses returned elevated rates of HTTP 503 responses.

The incident affected episode publishing, RSS feed availability, and downstream distribution to podcast directories. We restored service by rolling back the deployment, increasing worker capacity, and replaying queued feed-generation jobs.

## Customer Impact

Between 14:08 and 16:25 UTC:

- Approximately 31% of episode publishing requests were delayed.
- 18,420 newly published or updated episodes did not appear in RSS feeds at their scheduled time.
- Approximately 6.7% of uncached RSS feed requests returned HTTP 503 responses.
- Cached feeds remained available but could contain stale episode or metadata information.
- Podcast directories and listening applications experienced delayed ingestion of new episodes.
- Audio files already published before the incident remained available.
- No customer data or media files were lost.

Most delayed episodes appeared in feeds within 46 minutes of service restoration. The remaining backlog was fully processed by 17:11 UTC.

## Timeline

All times are in UTC on July 24, 2026.

- **13:52** — Version `feed-generator-2026.07.24.3` begins deployment to production.
- **14:03** — Deployment completes across all feed-generation workers.
- **14:08** — Feed-generation queue latency begins increasing.
- **14:12** — The first workers enter repeated restart loops after exceeding their memory limits.
- **14:17** — Automated monitoring alerts on worker restart rate. The alert is classified as SEV-2 because queue-age thresholds have not yet been crossed.
- **14:24** — Customer support receives the first report of a published episode missing from an RSS feed.
- **14:29** — Queue-age alert fires, and the incident is escalated to SEV-1.
- **14:34** — The on-call engineer confirms that existing feeds are mostly served from cache while newly generated feeds are failing or delayed.
- **14:41** — The team identifies unusually high memory consumption in feed-generation workers.
- **14:49** — Autoscaling reaches the configured worker limit but does not reduce queue growth because new workers also restart under affected workloads.
- **15:02** — Engineers correlate the failures with the deployment completed at 14:03.
- **15:11** — Rollback begins.
- **15:24** — Rollback completes. Worker restart rates return to normal.
- **15:31** — Queue growth stops, but approximately 2.1 million jobs remain pending.
- **15:38** — Worker capacity is temporarily increased, and replay processing begins.
- **15:57** — RSS feed error rates return to normal.
- **16:25** — Publishing latency falls below the incident threshold. The incident is marked resolved.
- **17:11** — The remaining backlog is fully processed.
- **18:05** — Temporary worker capacity is removed after confirming stable queue depth and memory utilization.

## Root Cause

The deployment introduced a new XML validation step intended to detect malformed episode metadata before publishing. The validator constructed an in-memory document containing the complete podcast feed and retained references to intermediate XML nodes until validation finished.

For typical feeds, the additional memory usage was small. For large feeds with thousands of episodes or unusually large metadata fields, peak memory consumption increased by four to seven times. These workloads exceeded the workers’ container memory limit, causing the operating system to terminate the processes.

Jobs interrupted by worker termination were automatically returned to the queue. Because retries were not bounded for this failure mode, the same large feeds were repeatedly assigned to new workers and caused additional restarts. This reduced effective processing capacity for all feed-generation jobs and allowed the queue to grow rapidly.

As cached feeds expired, requests requiring regeneration began timing out or returning HTTP 503 responses, expanding the incident from delayed publishing to partial feed unavailability.

## Contributing Factors

- Load testing used feeds with representative request volume but not representative feed size or metadata complexity.
- The deployment was evaluated primarily on average memory consumption rather than peak memory per job.
- Feed-generation jobs did not have a maximum retry count or a dead-letter queue.
- Autoscaling was based on queue depth and CPU usage, but not worker restart rate or memory pressure.
- The deployment proceeded to the full worker fleet before processing enough large-feed samples to expose the regression.
- The initial restart-rate alert was classified as SEV-2, delaying incident escalation by 12 minutes.
- Cache expiration was not extended automatically when feed-generation capacity became unhealthy.

## Resolution and Recovery

We rolled back the XML validation change and restarted the feed-generation worker fleet. After memory utilization stabilized, we temporarily increased worker capacity to process the backlog.

We also isolated repeatedly failing jobs so they could not monopolize workers during recovery. Once queue latency and feed error rates returned to normal, we replayed those jobs separately and verified that all scheduled publications were reflected in their RSS feeds.

## Detection

The incident was first detected by an automated alert for elevated worker restart rates, nine minutes after the production deployment completed. The alert did not immediately page the primary incident responder because the restart threshold was configured as a lower-severity condition.

The customer-visible publishing delay was detected later by a queue-age alert. Feed availability degradation was detected by HTTP error-rate monitoring after cached feeds began expiring.

Detection was slower than intended because no alert directly correlated deployment events, memory-related worker terminations, and increasing job retries.

## What Went Well

- The deployment system supported a complete rollback without database or schema changes.
- Existing cached feeds reduced the number of listeners affected during the first hour.
- The incident team identified the responsible deployment within 33 minutes of escalation.
- No audio, metadata, analytics, or customer configuration data was lost.
- Queue replay tooling allowed the backlog to be processed without republishing duplicate episodes.

## What Went Poorly

- Pre-production testing did not include sufficiently large or complex feeds.
- Retry behavior amplified a localized memory issue into a platform-wide processing backlog.
- Autoscaling added unhealthy workers without addressing the limiting resource.
- The initial alert severity did not reflect the risk posed by widespread worker restarts.
- Customer-facing status updates began 41 minutes after the first confirmed impact.
- Internal dashboards did not clearly distinguish newly queued work from repeatedly retried jobs.

## Action Items

| Action | Owner | Priority | Target Date |
|---|---|---:|---|
| Replace the in-memory XML validator with a streaming implementation and add explicit memory limits per validation operation. | Feed Platform | P0 | August 7, 2026 |
| Add large-feed and oversized-metadata fixtures to performance and regression test suites. | Quality Engineering | P0 | August 5, 2026 |
| Introduce bounded retries and a dead-letter queue for feed-generation jobs. | Messaging Platform | P0 | August 12, 2026 |
| Add a deployment canary stage that includes the largest 1% of production feed profiles using anonymized fixtures. | Release Engineering | P0 | August 14, 2026 |
| Page the primary on-call responder when worker restart rates exceed the fleet-wide threshold for five minutes. | Reliability Engineering | P1 | August 4, 2026 |
| Add dashboards for peak memory per job, retry amplification, and effective worker throughput. | Observability | P1 | August 10, 2026 |
| Configure autoscaling to account for memory pressure and unhealthy worker churn. | Infrastructure | P1 | August 19, 2026 |
| Extend stale feed serving during feed-generation incidents when a previously valid cached feed is available. | Delivery Platform | P1 | August 21, 2026 |
| Automate customer-status notifications when publishing latency exceeds 10 minutes platform-wide. | Incident Management | P2 | August 28, 2026 |
| Document and test a backlog-recovery procedure, including isolation of repeatedly failing jobs. | Reliability Engineering | P2 | August 18, 2026 |

## Prevention

Future feed-generation changes will be tested against both high-volume traffic and worst-case feed sizes before production deployment. Releases affecting parsing, serialization, or validation will use staged canaries with memory-based rollback criteria.

We are also changing the queue architecture so that a small number of pathological jobs cannot repeatedly consume the capacity required by healthy workloads. Combined with improved stale-cache behavior, these changes will reduce both the likelihood of a similar failure and its customer impact.