# Postmortem: Photo Library Upload and Search Outage **Incident date:** 2026-07-24 **Duration:** 2 hours 17 minutes **Severity:** SEV-1 **Status:** Resolved ## Summary On July 24, the self-hosted photo library became unavailable for 2 hours and 17 minutes. Users could not reliably upload, browse, or search their photos. Existing media files remained intact, but requests requiring database access returned errors or timed out. The incident began when a scheduled metadata reindex consumed the remaining disk space on the database volume. PostgreSQL stopped accepting writes, application workers entered an aggressive retry loop, and the resulting connection pressure prevented read-only requests from completing. Service was restored by stopping the reindex, expanding the volume, removing safe-to-delete temporary data, and restarting the database and application workers. ## Customer Impact During the incident: - The web and mobile applications frequently displayed loading errors. - Photo and video uploads failed or remained indefinitely in “processing.” - Albums, timelines, and search results were unavailable or incomplete. - Background thumbnail generation and metadata extraction stopped. - API clients received elevated HTTP 500 and 503 responses. Approximately 84% of active users encountered at least one failed request. Of 1,742 attempted uploads, 611 initially failed. Clients automatically retried most failures after recovery; 47 uploads required users to retry manually. No original photos or videos were lost. Twelve duplicate asset records created by client retries were identified and removed after recovery. ## Timeline All times are in EDT. - **09:00** — The weekly metadata reindex starts as scheduled. - **09:31** — Database volume utilization exceeds 90%. A warning alert is recorded but not delivered because the notification webhook still points to a retired endpoint. - **09:44** — Reindex temporary files consume the remaining free space. - **09:45** — PostgreSQL begins rejecting writes with `No space left on device`. - **09:47** — Upload processing failures increase sharply. Workers immediately retry failed jobs. - **09:50** — Database connection usage reaches its configured maximum. Browse and search requests begin timing out. - **09:56** — The first user reports that the photo timeline will not load. - **10:08** — The on-call engineer confirms widespread HTTP 500 and 503 responses. - **10:14** — The incident is declared SEV-1. Upload processing is paused. - **10:22** — The team identifies a full database volume and stops the metadata reindex. - **10:31** — Temporary reindex files and expired database log archives are removed, restoring 18 GB of free space. - **10:38** — PostgreSQL is restarted and passes consistency checks. - **10:45** — API services recover, but background workers again saturate database connections while draining the retry queue. - **10:52** — Worker concurrency is reduced and automatic retries are temporarily disabled. - **11:03** — Browse, album, and search error rates return to normal. - **11:18** — Upload processing resumes at reduced concurrency. - **11:39** — The backlog is draining normally, and the incident is marked resolved. - **14:20** — All queued uploads and background jobs have completed. - **16:10** — A reconciliation job confirms that no original media files were lost. ## Root Cause The immediate cause was exhaustion of disk space on the volume shared by PostgreSQL data, transaction logs, and temporary reindex files. The scheduled metadata reindex used a new query plan after a recent schema migration. The plan created substantially larger temporary files than previous runs. Capacity planning for the migration considered permanent index size but did not include worst-case temporary disk usage during index construction. When the volume filled, PostgreSQL could no longer write transaction and temporary data. Application workers treated these database failures as transient and retried immediately without exponential backoff or a retry limit. This retry storm exhausted the database connection pool and expanded the incident from degraded uploads to a near-total outage affecting reads, search, and authentication. ## Contributing Factors - Database data and temporary files shared the same volume. - The reindex ran during a period of normal user activity. - Disk-capacity estimates excluded temporary workspace requirements. - The 90% disk-usage alert was configured, but its notification destination was obsolete. - The application had no circuit breaker for repeated database write failures. - Background workers shared the same database connection pool as interactive requests. - The operational runbook did not document how to pause workers independently of the API. - The staging environment contained too little data to reproduce the production query plan or disk usage. ## Resolution and Recovery The response team stopped the reindex and paused background processing to prevent additional database load. Temporary reindex files and expired archived logs were removed after confirming that they were not required for recovery. The database volume was then expanded from 250 GB to 400 GB. After PostgreSQL restarted successfully, application traffic was restored in stages. Worker concurrency was limited while the retry queue drained, preserving database capacity for interactive requests. A reconciliation process compared object storage contents with database records and confirmed that all original media remained available. ## What Went Well - Original media was stored separately from the database volume and was unaffected. - Client-side upload retries recovered most failed uploads automatically. - Database consistency checks completed successfully after restart. - The team’s reconciliation tooling quickly verified media integrity. - Staged worker recovery prevented a second full outage. ## What Went Poorly - The disk alert did not reach the on-call engineer. - Initial symptoms appeared as unrelated upload, search, and authentication failures, delaying diagnosis. - Worker retries amplified the database failure. - Interactive and background workloads lacked resource isolation. - Recovery required manual commands that were not documented in the runbook. - There was no tested procedure for estimating reindex temporary-space requirements. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---:| | Move PostgreSQL temporary files to a dedicated volume with a size limit | Infrastructure | P0 | 2026-08-07 | | Replace and test all production alert notification endpoints | SRE | P0 | 2026-08-01 | | Add critical alerts at 85%, 90%, and 95% database-volume utilization | SRE | P0 | 2026-08-01 | | Add exponential backoff, jitter, and retry limits to background jobs | Backend | P0 | 2026-08-14 | | Reserve database connections for interactive API traffic | Backend | P0 | 2026-08-14 | | Add a circuit breaker that pauses workers after sustained database write failures | Backend | P1 | 2026-08-21 | | Schedule reindex operations during the maintenance window | Operations | P1 | 2026-08-07 | | Add preflight checks for free disk space before maintenance jobs | Infrastructure | P1 | 2026-08-14 | | Build a production-scale dataset for migration and reindex testing | Platform | P1 | 2026-09-04 | | Document and test procedures for pausing workers and draining queues | SRE | P1 | 2026-08-14 | | Add automated object-storage-to-database integrity checks | Data | P2 | 2026-09-18 | | Review capacity forecasts quarterly and before schema migrations | Engineering | P2 | 2026-09-30 | ## Lessons Learned A storage-capacity issue became a full-service outage because maintenance workloads, background jobs, and user-facing requests shared the same constrained resources. Additional disk capacity reduces immediate risk, but the durable fixes are workload isolation, bounded retries, verified alert delivery, and production-scale testing of maintenance operations.