# Postmortem: Photo Library Unavailable During Storage Migration

**Incident date:** July 24, 2026  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 24, the self-hosted photo library became unavailable during a migration of its media storage volume. The application remained online, but requests for albums, thumbnails, and original files failed because the media directory was mounted at the wrong path after the storage host restarted.

A recovery script then worsened the incident by creating an empty directory at the expected mount point. This allowed the application to start successfully while masking the missing storage mount. Service was restored after the incorrect directory was removed, the storage volume was mounted at the expected path, and thumbnail workers were restarted.

No original photos or metadata were lost.

## Customer Impact

From 19:18 to 21:05 EDT:

- All users received errors when opening albums or viewing photos.
- Mobile uploads appeared to complete but remained queued and were not written to permanent media storage.
- Shared album links returned HTTP 500 responses.
- Background thumbnail generation failed and retried repeatedly.
- Search results remained available because metadata was stored separately, but selecting a result failed to load the image.

There were 286 failed page requests, 41 delayed uploads, and 3,912 failed thumbnail jobs. All delayed uploads were processed after service recovery.

## Timeline

All times are in EDT.

- **18:52** — Administrator begins migrating the media volume from a local disk to a larger network-attached storage volume.
- **19:07** — Initial file synchronization completes. A final incremental synchronization is scheduled after application shutdown.
- **19:12** — Photo library services are stopped, and the final synchronization begins.
- **19:16** — Synchronization completes without reported errors.
- **19:17** — Storage host is restarted to validate automatic mounting.
- **19:18** — Application containers restart before the network storage mount becomes available.
- **19:18** — Startup script creates `/srv/photos/library` as an empty local directory because the expected path does not exist.
- **19:19** — Application health checks pass because they verify only HTTP responsiveness and database connectivity.
- **19:23** — First user reports that albums contain broken images.
- **19:25** — Monitoring detects an increase in HTTP 500 responses but does not page because the alert threshold requires 15 consecutive minutes of failures.
- **19:31** — Thumbnail workers enter a retry loop, increasing CPU usage on the application host.
- **19:34** — Administrator begins investigating the application and database logs.
- **19:46** — Database integrity is confirmed; attention shifts to file permissions because media requests report “file not found.”
- **20:02** — Permissions on the empty local directory are changed. This does not resolve the incident.
- **20:14** — Administrator notices that the media directory contains no files and confirms the network volume is not mounted.
- **20:21** — Manual mount attempt fails because the startup-created directory is in use by running containers.
- **20:29** — Application and worker containers are stopped.
- **20:34** — Empty local directory is removed, and the network storage volume is mounted successfully.
- **20:39** — File counts and sampled checksums are compared with the migration source; no data loss is found.
- **20:47** — Application containers restart with the correct media volume.
- **20:51** — Album and original-file requests recover.
- **20:56** — Thumbnail workers restart with reduced concurrency to prevent a retry surge.
- **21:05** — Upload processing and thumbnail queues return to normal. Incident is declared resolved.

## Root Cause

The immediate cause was a startup ordering failure between the application containers and the network storage mount. The storage host had not completed mounting the new volume when the container runtime started the photo library.

The application startup script contained the following behavior:

1. Check whether `/srv/photos/library` exists.
2. Create it if it does not exist.
3. Start the application.

Because the network volume was not mounted, the script created a normal directory on the application host at the expected mount point. The application then started against that empty directory. Its health check passed because it tested only the HTTP endpoint and database connection, not the availability or contents of media storage.

## Contributing Factors

- The migration procedure did not include a tested rollback plan.
- Container startup was not explicitly dependent on the storage mount.
- The storage mount used a network target that became available later in the boot sequence.
- Health checks did not verify that the media volume was mounted or writable.
- The expected media directory could be created automatically, masking mount failures.
- Alerting focused on total application availability rather than media retrieval success.
- Troubleshooting initially focused on database integrity and permissions, delaying identification of the missing mount.
- Thumbnail workers retried failures without sufficient backoff, increasing load and log volume.

## Resolution

Service was restored by stopping the application containers, removing the empty local mount-point directory, mounting the network storage volume at `/srv/photos/library`, and restarting the application.

The team verified recovery by:

- Comparing file counts between the migration source and destination.
- Validating checksums for a sample of original files.
- Opening recent and historical albums.
- Uploading and retrieving a new image.
- Processing the delayed upload queue.
- Monitoring thumbnail generation until the retry backlog cleared.

## Data Integrity

No photos, videos, or metadata were lost. Uploads submitted during the incident remained in temporary application storage and were processed after recovery.

The empty local directory contained no customer media and was removed before the network volume was mounted.

## What Went Well

- The original storage volume remained unchanged and available for rollback.
- Upload staging prevented queued uploads from being lost.
- Database backups and integrity checks were current.
- File-count and checksum verification provided confidence before reopening the service.
- Users reported the issue quickly, reducing time to detection.

## What Went Poorly

- Application health checks reported a healthy service while its primary function was unavailable.
- The startup script converted a missing dependency into an apparently valid empty directory.
- Monitoring did not immediately page on widespread media failures.
- The migration was performed without a full restart test in a staging environment.
- Worker retries added noise and resource pressure during diagnosis.

## Action Items

| Action | Owner | Priority | Due date | Status |
|---|---|---:|---|---|
| Add a startup check that verifies the media path is an active mount and contains a known sentinel file | Platform | P0 | August 2, 2026 | In progress |
| Configure the application service to start only after the network storage mount is available | Platform | P0 | August 2, 2026 | In progress |
| Remove automatic creation of the production media directory from the startup script | Application | P0 | August 1, 2026 | Complete |
| Update health checks to read a known media object and verify write access to the upload directory | Application | P1 | August 7, 2026 | Planned |
| Page when media retrieval failures exceed 5% for five minutes | Operations | P1 | August 5, 2026 | Planned |
| Add exponential backoff and a retry limit to thumbnail jobs | Application | P1 | August 14, 2026 | Planned |
| Document storage migration, validation, and rollback procedures | Operations | P1 | August 9, 2026 | Planned |
| Add a reboot test to the storage migration checklist | Operations | P1 | August 9, 2026 | Planned |
| Run quarterly restore and storage-failure exercises | Operations | P2 | September 30, 2026 | Planned |
| Add disk-usage alerts for the local filesystem beneath network mount points | Platform | P2 | August 21, 2026 | Planned |

## Prevention

Future deployments will treat media storage as a required dependency rather than an optional directory. The application will refuse to start unless it can confirm that the expected volume is mounted, readable, writable, and contains a deployment-specific sentinel file.

Operational procedures will also require a controlled reboot and end-to-end media test before any storage migration is considered complete.