# Postmortem: Photo Library Upload and Search Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 24, the self-hosted photo library became unavailable after its PostgreSQL database exhausted disk space. A background metadata migration generated significantly more temporary data than expected while thumbnail workers continued writing new files to the same volume.

Once the volume reached capacity, database writes failed and PostgreSQL entered crash recovery. The web application returned errors for uploads, searches, album changes, and most authenticated requests. Existing original photos were not lost, but 43 uploads accepted immediately before the failure had incomplete database records and required recovery from the upload staging directory.

Service was restored by stopping background workers, expanding the volume, clearing safe-to-delete temporary files, and allowing PostgreSQL recovery to complete.

## Customer Impact

During the incident:

- The web and mobile clients returned `500` or `503` errors for most operations.
- New photo and video uploads failed or remained indefinitely in “processing.”
- Search, facial recognition results, and album updates were unavailable.
- Shared links intermittently failed because access checks required database queries.
- Previously cached thumbnails remained visible in some clients.
- Original media files already stored in the library were unaffected.
- Forty-three newly uploaded assets required reconciliation; 41 were recovered automatically, and 2 had to be uploaded again.
- No confirmed unauthorized access or permanent loss of previously indexed media occurred.

## Timeline

All times are in EDT.

- **09:02** — A metadata migration begins after the scheduled application upgrade.
- **09:18** — Database temporary-file usage rises above its normal range. No alert fires because monitoring covers total free space only at 10% and 5% thresholds.
- **09:41** — Thumbnail workers begin processing a recently imported archive of approximately 180,000 photos.
- **10:07** — Free space falls below 10%. The warning notification is sent to an obsolete email address.
- **10:23** — PostgreSQL logs its first `No space left on device` error while writing a temporary query file.
- **10:24** — Uploads begin failing. API error rates rise from below 0.2% to 68%.
- **10:27** — The database process restarts and enters crash recovery. The application remains online and repeatedly attempts connections.
- **10:31** — The administrator is notified by users through a community chat channel.
- **10:38** — The incident is declared SEV-1. Upload and thumbnail workers are stopped.
- **10:49** — Investigation confirms the shared data volume is at 100% capacity.
- **11:02** — Temporary migration files and regenerable thumbnail caches are identified as safe to remove.
- **11:14** — Enough space is freed for PostgreSQL to complete crash recovery.
- **11:29** — The database becomes available, but write latency remains elevated.
- **11:36** — The underlying volume is expanded from 2 TB to 3 TB.
- **11:48** — API error rates return to normal. Uploads remain disabled while consistency checks run.
- **12:09** — Upload reconciliation finds 43 files without complete asset records.
- **12:21** — Forty-one incomplete uploads are recovered from staging.
- **12:41** — Uploads and background processing are re-enabled with reduced concurrency.
- **12:44** — The incident is resolved.
- **14:12** — Two unrecoverable partial uploads are identified, and the affected user is asked to upload them again.

## Root Cause

The immediate cause was exhaustion of the volume shared by PostgreSQL, uploaded media, thumbnails, and application temporary files.

A metadata migration performed a large table rewrite and created temporary files proportional to the database size. Its storage requirements had been tested against a smaller library but not against the production dataset. At the same time, thumbnail workers processed a large import and consumed the remaining free space.

PostgreSQL could not complete writes after the filesystem reached capacity. The database restarted and entered crash recovery, making the application unavailable.

## Contributing Factors

- Database files, original media, thumbnails, staging files, and temporary migration data shared one volume.
- Capacity planning did not include migration working space or concurrent thumbnail generation.
- The upgrade procedure did not define a minimum free-space requirement.
- Background workers remained enabled during the migration.
- Disk alerts were configured only for percentage thresholds, which provided little warning on a multi-terabyte volume.
- The warning alert was routed to an obsolete email address and had no secondary notification channel.
- Application health checks verified that the process was running but did not verify database write capability.
- Clients received generic server errors and continued retrying, increasing connection pressure during recovery.
- Upload staging had no documented reconciliation procedure.

## Resolution and Recovery

Responders stopped all background workers to prevent additional writes, removed regenerable thumbnail cache files, and expanded the affected volume. PostgreSQL then completed crash recovery successfully.

After database availability was restored, integrity checks were run against the asset and album tables. Files in upload staging were compared with database records, allowing 41 of 43 incomplete uploads to be recovered. Background jobs were re-enabled at reduced concurrency and monitored until their queues stabilized.

## What Went Well

- Original media files and existing database records remained intact.
- Upload staging preserved most files that had not yet been committed to the database.
- Thumbnail caches could be safely removed to create recovery space.
- Database backups had completed successfully before the incident and were available if recovery failed.
- Once the failure was identified, responders restored core service without needing a full database restore.

## What Went Poorly

- Monitoring detected the condition but did not notify an active responder.
- The application remained nominally healthy while its primary dependency was unavailable.
- Migration resource requirements were not documented or validated against production scale.
- Recovery depended on manual knowledge of which directories were safe to clear.
- Users learned about the outage through failed requests rather than a service-status notice.
- The upload API acknowledged some files before their database transactions were durably committed.

## Action Items

| Action | Owner | Priority | Due date | Status |
|---|---|---:|---|---|
| Move PostgreSQL to a dedicated volume with reserved recovery capacity | Infrastructure | P0 | August 7, 2026 | In progress |
| Add absolute free-space and projected-exhaustion alerts | Infrastructure | P0 | August 3, 2026 | In progress |
| Route critical alerts to the on-call notification channel and test monthly | Operations | P0 | August 2, 2026 | Complete |
| Add a pre-upgrade check requiring sufficient migration working space | Backend | P0 | August 14, 2026 | Planned |
| Pause thumbnail, indexing, and recognition workers during storage-intensive migrations | Backend | P1 | August 14, 2026 | Planned |
| Add database read/write probes to application readiness checks | Platform | P1 | August 10, 2026 | Planned |
| Limit client retries using exponential backoff and jitter | Client | P1 | August 21, 2026 | Planned |
| Make upload completion dependent on durable database commit | Backend | P1 | August 21, 2026 | Planned |
| Automate reconciliation of staged uploads after interrupted processing | Backend | P1 | August 28, 2026 | Planned |
| Document disk-exhaustion recovery and safe cache cleanup procedures | Operations | P2 | August 7, 2026 | In progress |
| Run production-scale migration tests using anonymized dataset statistics | Release Engineering | P2 | September 4, 2026 | Planned |
| Add an administrator-visible maintenance and incident banner | Web | P2 | September 11, 2026 | Planned |

## Lessons Learned

Storage exhaustion must be treated as a predictable capacity event rather than an exceptional failure. Percentage-based alerts alone were insufficient, and colocating critical database files with large, regenerable media derivatives allowed background work to take down the entire service.

Future upgrades will include production-scale storage estimates, enforced preflight checks, and explicit control of background workloads. Separating database storage and improving upload durability will reduce both the likelihood and impact of a similar incident.