# Postmortem: Photo Library Upload and Browsing Outage **Incident date:** July 24, 2026 **Duration:** 1 hour 47 minutes **Severity:** SEV-1 **Status:** Resolved **Services affected:** Web application, mobile uploads, thumbnail generation, search **Authors:** Photo Platform Team ## Summary On July 24, the self-hosted photo library became unavailable after the primary PostgreSQL database exhausted its available disk space. A newly deployed metadata backfill generated substantially more database write-ahead log (WAL) data than expected. At the same time, a stale replication slot prevented old WAL segments from being removed. Once the database volume reached 100% utilization, PostgreSQL rejected writes and later entered crash recovery. API requests that required database access began timing out, upload workers repeatedly retried failed transactions, and the resulting connection pressure made read-only operations such as browsing albums unreliable. The team restored service by stopping the backfill and upload workers, removing the unused replication slot, expanding the database volume, and restarting PostgreSQL. No original photo or video files were lost. Twenty-three uploads initiated during the incident had to be retried by users. ## Customer Impact From 14:18 to 16:05 EDT: - The web application failed to load libraries or albums for most users. - Mobile clients could not complete photo and video uploads. - Search, face recognition results, and recently added views were unavailable. - Shared album links intermittently returned `502` or `504` responses. - Background thumbnail and metadata processing stopped. - Twenty-three uploads failed after clients exhausted their retry limits. - Forty-one uploads were delayed and completed automatically after recovery. - No original media files already stored in the library were deleted or corrupted. - No evidence of unauthorized access or data exposure was found. ## Detection The incident was first reported by a user at 14:21 EDT. Infrastructure alerts did not fire before the outage because disk monitoring covered the host filesystem but not the separately mounted database volume. An existing PostgreSQL health check detected connection failures at 14:25, but it was configured as an informational alert and did not page the on-call engineer. ## Timeline All times are in EDT. - **13:42** — Version `2026.07.24-1` is deployed. The release includes a background backfill to extract additional camera metadata from existing assets. - **13:49** — The backfill starts processing approximately 1.8 million asset records. - **14:03** — Database disk utilization exceeds 90%. No alert is generated. - **14:12** — A batch of large video uploads begins, increasing database and object-storage activity. - **14:18** — PostgreSQL reports `No space left on device` while extending a WAL segment. API write requests begin failing. - **14:20** — Upload workers start retrying failed transactions, increasing database connection usage. - **14:21** — The first user reports that the library is not loading. - **14:25** — The database health check fails, generating a non-paging notification. - **14:29** — The on-call engineer begins investigating elevated API latency. - **14:34** — The team identifies that the PostgreSQL volume is at 100% utilization and that WAL files account for most recent growth. - **14:39** — The metadata backfill and upload workers are stopped to reduce writes and connection pressure. - **14:46** — Investigation finds an inactive replication slot left behind by a retired backup replica. The slot has retained 186 GB of WAL. - **14:52** — The team confirms that the replica associated with the slot was permanently decommissioned and removes the stale slot. - **14:57** — PostgreSQL cannot immediately reclaim enough space because active processes still hold deleted WAL files open. - **15:04** — The database volume is expanded from 500 GB to 750 GB. - **15:13** — PostgreSQL is restarted and begins crash recovery. - **15:31** — Crash recovery completes. Database connections succeed, but API latency remains elevated while caches warm. - **15:38** — Read traffic is restored. Album browsing and shared links begin recovering. - **15:44** — Upload workers are restarted with concurrency reduced from 16 to 4. - **15:52** — Thumbnail and metadata workers resume. - **16:05** — Error rates and latency return to normal. The incident is resolved. - **17:20** — Upload reconciliation identifies 23 client uploads that did not complete and 41 delayed uploads that recovered automatically. - **19:10** — Integrity checks confirm that existing originals and generated assets are intact. ## Root Cause The immediate cause of the outage was exhaustion of the PostgreSQL data volume. A metadata backfill introduced in the deployment updated most asset rows, producing a sustained increase in WAL generation. Under normal conditions, PostgreSQL would recycle older WAL segments. However, a physical replication slot associated with a retired backup replica had remained active for 47 days. PostgreSQL therefore retained WAL segments that the missing replica had not acknowledged. The combination of elevated WAL generation and indefinite WAL retention consumed the remaining 214 GB on the database volume. PostgreSQL could no longer extend WAL files or complete transactions, causing write failures and crash recovery. ## Contributing Factors - Disk alerts monitored the host root filesystem but not the mounted PostgreSQL volume. - Replication slot lag was visible in metrics but had no alert threshold. - The retired backup replica was removed without deleting its replication slot. - The backfill had been tested on a much smaller dataset and lacked a production-scale estimate of WAL growth. - The backfill had no automatic pause condition based on disk utilization or database health. - Upload workers retried database failures without sufficient exponential backoff, increasing connection pressure. - Read and write API traffic shared the same database connection pool, so write failures degraded browsing. - The database volume had only 30% free space before the deployment, below the intended 40% operating margin. - The database health notification was not configured to page the on-call engineer. ## Resolution and Recovery The team stopped write-heavy background work, removed the obsolete replication slot, and expanded the database volume. PostgreSQL was then restarted to release file handles and complete crash recovery. After database availability was restored, services were restarted gradually. Upload concurrency was temporarily reduced to prevent a retry surge. Background jobs were resumed only after API latency and disk utilization stabilized. A reconciliation job compared upload sessions, database records, and object-storage keys. It confirmed that existing media was intact and identified uploads requiring client retries. ## What Went Well - Original media was stored separately from the database and was not affected by database disk exhaustion. - The team quickly identified WAL retention as the primary source of disk growth. - Upload session records allowed delayed and incomplete uploads to be distinguished reliably. - Database checksums and application-level integrity checks found no corruption. - Reducing worker concurrency prevented a second overload during recovery. ## What Went Poorly - Monitoring did not cover the filesystem whose exhaustion caused the outage. - A database health failure did not page the on-call engineer. - Replica decommissioning was not an automated or documented process. - The migration lacked rate limits, safety thresholds, and a fast operational kill switch. - Retry behavior amplified load during a database failure. - The status page was updated 36 minutes after customer impact began. ## Where We Got Lucky - The database failed before object-storage capacity was affected. - The stale replication slot belonged to a replica that was confirmed to be permanently retired. - Crash recovery completed without database corruption. - Most mobile clients retained their local files and could retry uploads. ## Action Items | Action | Owner | Priority | Due date | Status | |---|---|---:|---|---| | Add paging alerts at 80%, 90%, and 95% utilization for every mounted data volume | Infrastructure | P0 | July 27, 2026 | Complete | | Alert when any replication slot retains more than 10 GB of WAL or is inactive for more than 24 hours | Database | P0 | July 27, 2026 | Complete | | Change database health-check failures from informational notifications to paging alerts | SRE | P0 | July 26, 2026 | Complete | | Add automatic backfill suspension when database disk utilization exceeds 85% | Photo Platform | P0 | August 7, 2026 | In progress | | Add bounded exponential backoff and jitter to upload-worker database retries | Photo Platform | P1 | August 7, 2026 | In progress | | Separate interactive API and background-worker database connection pools | Photo Platform | P1 | August 14, 2026 | Planned | | Require capacity and WAL-growth estimates for large production backfills | Engineering Productivity | P1 | August 14, 2026 | Planned | | Automate replication-slot removal as part of replica decommissioning | Database | P1 | August 21, 2026 | Planned | | Add a weekly audit for inactive replication slots and unconsumed WAL | Database | P1 | August 7, 2026 | In progress | | Maintain at least 40% free space on the primary database volume | Infrastructure | P1 | August 7, 2026 | In progress | | Add upload reconciliation to the standard incident recovery runbook | Photo Platform | P2 | August 21, 2026 | Planned | | Update the incident process to require a status-page update within 15 minutes of confirmed customer impact | SRE | P2 | August 14, 2026 | Planned | ## Lessons Learned Database free space must be treated as a reliability boundary, especially for self-hosted deployments where storage does not expand automatically. Background migrations also need production-scale resource estimates and runtime safeguards rather than relying solely on staging tests. Replication slots are durable database state and must be managed as part of a replica’s lifecycle. Removing a replica without removing or monitoring its slot allowed an otherwise recoverable workload increase to become a full outage. Finally, retry behavior must assume that dependencies can remain unavailable. Unbounded or synchronized retries can turn a localized failure into broader service degradation and make recovery slower.