# Postmortem: Photo Library Unavailable Following Storage Migration

**Incident date:** 2026-07-18  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 18, the self-hosted photo library became unavailable after a scheduled migration of its media volume to a larger storage array. The application initially appeared healthy, but requests for thumbnails and original files stalled. The resulting backlog exhausted the application’s database connection pool, causing uploads, searches, album views, and authentication requests to fail.

The immediate cause was an incorrect ownership mapping on the migrated media volume. A background thumbnail worker repeatedly retried files it could read but could not update. Our health checks did not exercise media writes, so the deployment was reported as healthy while the service was already degrading.

We restored service by stopping the worker queue, correcting the volume ownership, restarting the application, and gradually draining queued jobs. No original photos were lost or corrupted.

## Customer Impact

During the incident:

- The web and mobile applications returned errors or loaded indefinitely.
- Existing photos and thumbnails were intermittently unavailable.
- New uploads failed or remained stuck in processing.
- Search, albums, and shared links were unavailable.
- Automated mobile backups retried repeatedly, increasing request volume.

Approximately 86% of requests made during the incident failed or exceeded the 30-second client timeout. Forty-three uploads were interrupted; all were retried successfully after recovery. No original media files, albums, or metadata were lost.

## Timeline

All times are in Eastern Daylight Time.

- **09:00** — Scheduled storage migration begins. The application is placed in maintenance mode.
- **09:18** — Media files are copied to the new storage array using an administrator account.
- **09:34** — The new volume is mounted and the application containers are restarted.
- **09:37** — HTTP and database health checks pass. Maintenance mode is disabled.
- **09:41** — Thumbnail workers begin reporting permission errors while replacing generated files.
- **09:45** — Worker retry volume increases. No alert fires because jobs are still being acknowledged and retried.
- **09:53** — Users begin reporting missing thumbnails and stalled uploads.
- **10:02** — Database latency alert fires after the connection pool reaches saturation.
- **10:08** — On-call engineer acknowledges the alert and begins investigating the database.
- **10:21** — Database health is confirmed; application logs show requests waiting on media-processing transactions.
- **10:29** — Engineers identify a rapidly growing thumbnail job backlog and pause all workers.
- **10:35** — API availability partially recovers, but original image requests continue to fail.
- **10:44** — Permission errors are traced to numeric user and group IDs on the new volume.
- **11:02** — Ownership and directory permissions are corrected across the media volume.
- **11:17** — Read and write checks succeed from inside the application and worker containers.
- **11:24** — Application containers are restarted. Core browsing and upload functionality recovers.
- **11:31** — Workers are restarted with reduced concurrency.
- **11:47** — Error rate returns to baseline. Previously interrupted uploads begin retrying successfully.
- **12:24** — Worker backlog is fully drained. Incident is declared resolved.

## Root Cause

The new storage array was prepared by an administrator account and preserved numeric ownership values that did not match the user ID used by the application containers.

The application process could read most original media files because they were world-readable. However, thumbnail workers could not replace temporary thumbnail files or update some generated-media directories. Each failed job was retried automatically with exponential backoff.

A thumbnail-processing job holds a database transaction while writing generated media and updating its metadata. As retrying jobs accumulated, worker processes occupied nearly all available database connections. Interactive application requests then waited for connections until they timed out, turning a media-volume permission issue into a full application outage.

## Contributing Factors

- The migration procedure verified that media files could be read but did not verify writes, renames, or deletions from inside the application containers.
- Health checks covered HTTP responses and database connectivity but did not exercise the media storage path.
- Worker retry metrics were not included in alerting.
- Background workers shared the same database connection pool as interactive requests.
- Worker concurrency was configured for normal operation and amplified the connection exhaustion during repeated failures.
- The rollback procedure had not been tested with the new mount layout.
- Maintenance mode was disabled before a representative upload and thumbnail-generation test completed.

## Resolution and Recovery

We paused background workers to release database connections and restore interactive application capacity. We then changed ownership of the media and generated-media directories to the application’s configured user and group IDs and verified file operations from inside each relevant container.

After restarting the application, we re-enabled workers at reduced concurrency and monitored storage errors, database utilization, and queue depth while the backlog drained. Interrupted uploads were reconciled against the database and object checksums; no missing or corrupted originals were found.

## What Went Well

- Original media and metadata remained intact.
- Database monitoring identified connection saturation before the database itself became unstable.
- Pausing workers restored partial service quickly.
- Upload clients retained interrupted files and retried them after recovery.
- Checksums made it possible to confirm data integrity without manually inspecting the entire library.

## What Went Poorly

- Initial investigation focused on the database rather than the worker queue, delaying identification of the storage error.
- Existing health checks produced a false sense of readiness after the migration.
- Permission errors were visible in logs but did not generate an alert.
- Interactive and background workloads lacked resource isolation.
- The migration runbook did not specify required numeric user and group IDs.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---:|
| Add a readiness check that creates, renames, reads, and deletes a file on each required media path | Platform | P0 | 2026-07-22 |
| Alert on sustained worker retries, dead-lettered jobs, and queue age | Observability | P0 | 2026-07-24 |
| Reserve database connections for interactive requests and cap worker pool usage | Application | P0 | 2026-07-25 |
| Update the storage migration runbook with container UID/GID requirements and validation commands | Operations | P1 | 2026-07-23 |
| Require an end-to-end upload, thumbnail, download, and deletion test before leaving maintenance mode | Operations | P1 | 2026-07-23 |
| Add a retry limit and dead-letter queue for persistent media-processing failures | Application | P1 | 2026-08-05 |
| Add circuit breaking that pauses workers when storage write failures exceed a threshold | Application | P1 | 2026-08-12 |
| Test and document storage rollback procedures in a staging environment | Platform | P1 | 2026-08-14 |
| Add a dashboard correlating storage errors, worker backlog, and database pool utilization | Observability | P2 | 2026-08-07 |
| Run quarterly restore and storage-migration exercises | Operations | P2 | 2026-09-30 |

## Lessons Learned

A successful mount and a readable file tree are insufficient validation for stateful media applications. Storage migrations must be tested from the same runtime identity and execution environment used in production, including all file operations performed by background jobs.

Background processing must also be isolated so that a failure in thumbnail generation cannot consume the resources required for authentication, browsing, and uploads.