# Postmortem: Photo Library Upload and Playback Outage

**Incident date:** July 14, 2026  
**Duration:** 2 hours 37 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 14, the photo library became unavailable after a routine storage maintenance operation. Users could not upload new photos, browse portions of their libraries, or stream videos. Existing thumbnails remained visible when served from cache, which initially made the incident appear limited to uploads.

The outage began when the storage server remounted the primary media filesystem as read-only following an I/O timeout. Application workers continued retrying failed writes while the media indexer opened additional database connections. This exhausted the database connection pool and caused requests unrelated to storage writes, including authentication and library browsing, to fail.

Service was restored by stopping background workers, remounting the media filesystem after verifying its integrity, restarting the database and application services, and gradually re-enabling queued jobs.

## Customer Impact

During the incident:

- 100% of photo and video uploads failed or remained indefinitely in a processing state.
- Approximately 82% of library and album requests returned errors or timed out.
- Video playback failed unless the requested rendition was already cached.
- Cached thumbnails remained available for some users, but opening the underlying asset failed.
- Mobile clients repeatedly retried uploads, increasing load and battery usage.
- No original media files were lost.
- 1,846 upload records were left incomplete; 1,791 were recovered automatically from client retries, and 55 required metadata cleanup before successful reprocessing.

The incident affected all 312 active household accounts hosted on the instance.

## Timeline

All times are in EDT.

| Time | Event |
|---|---|
| 09:02 | Scheduled storage scrub begins on the media server. |
| 09:11 | The storage controller records repeated timeouts from one disk in the primary media array. |
| 09:13 | The operating system remounts the media filesystem as read-only to prevent further corruption. |
| 09:14 | Upload workers begin logging write failures and retrying jobs. No alert fires because workers remain healthy at the process level. |
| 09:19 | The media indexer increases database connection usage while repeatedly retrying failed asset updates. |
| 09:24 | Database connection utilization reaches 100%. API latency and error rates rise sharply. |
| 09:27 | The first user report notes that mobile uploads are stuck. |
| 09:31 | The on-call engineer acknowledges the API error-rate alert and begins investigating the database. |
| 09:39 | The database is restarted after connection exhaustion is initially attributed to a leaked application connection. Error rates briefly decrease. |
| 09:44 | Background workers reconnect and exhaust the connection pool again. |
| 09:51 | The incident is escalated to SEV-1. Upload and indexing workers are stopped. |
| 10:03 | Engineers identify the read-only media filesystem as the common cause of worker failures. |
| 10:18 | Storage diagnostics identify a degraded disk path rather than filesystem corruption. The affected disk is isolated. |
| 10:31 | A filesystem check completes without finding structural errors. |
| 10:38 | The media filesystem is remounted read-write. Test reads, writes, and deletions succeed. |
| 10:44 | The database and API services are restarted. Browsing and authentication recover. |
| 10:52 | Upload workers are re-enabled at 10% concurrency. |
| 11:07 | Upload success rates return to normal. Indexing workers are re-enabled with retry limits. |
| 11:29 | The queued job backlog is declining, and video playback is fully restored. |
| 11:39 | The incident is resolved. Recovery monitoring continues for four hours. |
| 15:26 | Cleanup of incomplete upload records is completed. |

## Root Cause

The primary media filesystem was remounted as read-only after the storage subsystem encountered repeated I/O timeouts during a scheduled scrub. The immediate hardware issue was a degraded disk connection that intermittently stopped responding under sustained read load.

The application did not treat a read-only media volume as a terminal infrastructure failure. Upload and indexing workers retried write operations without a maximum retry count or meaningful backoff. Each retry opened a database transaction before attempting the filesystem write. When the write failed, the worker delayed inside the transaction while preparing the retry.

As the number of failed jobs increased, workers occupied every available database connection. The API then lost access to the connection pool, expanding a storage write failure into a service-wide outage.

## Contributing Factors

- Storage health checks verified that the mount path existed but did not test whether it was writable.
- Worker liveness checks only confirmed that processes were running.
- Filesystem writes occurred after database transactions had already been opened.
- Retry behavior had no global circuit breaker, maximum attempt count, or sufficient exponential backoff.
- Upload, indexing, and interactive API traffic shared the same database connection pool.
- The scheduled scrub ran during a period of normal user activity.
- Alerts monitored process availability and host capacity but not filesystem mount mode.
- The initial database restart temporarily masked the connection-exhaustion symptom and delayed identification of the storage failure.

## Resolution and Recovery

Engineers stopped upload and indexing workers to release database connections and prevent further retry amplification. After isolating the degraded disk path, they verified filesystem integrity and restored the media volume to read-write mode.

The database and API were restarted before background processing resumed. Workers were then enabled incrementally with reduced concurrency while error rates, storage latency, and database connections were monitored.

Incomplete upload records were reconciled against files present in temporary storage. Records with valid temporary files were requeued, while empty records were removed. No original assets were deleted or overwritten.

## What Went Well

- Filesystem protection prevented writes during an unstable storage condition.
- Original media and temporary upload data remained intact.
- Stopping background workers quickly restored database capacity.
- Incremental worker recovery avoided a second overload while processing the backlog.
- Client-side retry behavior allowed most incomplete uploads to recover automatically.

## What Went Poorly

- Monitoring detected the database symptom rather than the underlying storage condition.
- Background work could consume all database connections needed by interactive requests.
- Retry logic amplified a persistent infrastructure failure.
- The runbook emphasized database troubleshooting and did not include read-only mount checks.
- Users received generic server errors and had no indication that uploads were safely queued for retry.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add a readiness check that performs a write-and-delete probe on the media volume. | Platform | P0 | July 18, 2026 |
| Page when the media filesystem becomes read-only or storage I/O errors exceed threshold. | Platform | P0 | July 18, 2026 |
| Move filesystem writes outside database transactions and keep transaction duration bounded. | Application | P0 | July 25, 2026 |
| Add exponential backoff, retry limits, and dead-letter handling to upload and indexing jobs. | Application | P0 | July 25, 2026 |
| Implement a circuit breaker that pauses media workers after repeated storage write failures. | Application | P0 | July 25, 2026 |
| Reserve a dedicated database connection pool for interactive API traffic. | Database | P1 | August 1, 2026 |
| Change worker health checks to detect stalled queues and repeated job failures. | Platform | P1 | August 1, 2026 |
| Add an automated reconciliation tool for incomplete upload records and temporary files. | Application | P1 | August 8, 2026 |
| Move storage scrubs to the lowest-traffic maintenance window and reduce scrub I/O priority. | Operations | P1 | July 20, 2026 |
| Replace the degraded disk cable and run an extended storage burn-in test. | Operations | P0 | Completed |
| Update the outage runbook with storage mount, kernel log, and worker-amplification checks. | Reliability | P1 | July 22, 2026 |
| Display a specific retryable error in web and mobile clients when media storage is unavailable. | Client | P2 | August 15, 2026 |

## Lessons Learned

A dependency can appear healthy enough to mount and serve cached reads while still being unusable for the application’s core workload. Readiness checks must validate the operation the service actually requires, including writes.

Background processing also requires explicit isolation and failure boundaries. Unbounded retries converted a partial storage outage into a full application outage by exhausting a shared database resource. Future changes will ensure that persistent storage failures stop affected workers quickly while preserving capacity for browsing, authentication, diagnostics, and recovery.