# Postmortem: Photo Library Upload and Browsing Outage

**Incident date:** July 14, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 14, the self-hosted photo library became unavailable after the primary PostgreSQL database exhausted its disk space. A recently deployed metadata backfill generated significantly more database write-ahead log (WAL) data than expected. At the same time, a failed backup job prevented WAL segments from being archived and removed.

Once the database volume reached 100% utilization, PostgreSQL stopped accepting writes and entered crash recovery. The API returned errors for uploads, album changes, searches, and most photo-library views. Existing image files remained intact on object storage.

Service was restored by stopping the backfill, removing obsolete temporary backup files, expanding the database volume, and restarting PostgreSQL. No original photos or videos were lost. Some metadata changes made shortly before the outage had to be replayed from application logs.

## Customer Impact

During the incident:

- The web and mobile applications were unavailable or repeatedly displayed loading errors.
- New photo and video uploads failed.
- Album creation, sharing, favorites, captions, and deletion requests failed.
- Background thumbnail generation and facial-recognition jobs stopped.
- Public share links returned HTTP 500 responses.
- Twelve uploads reported success to clients but were not committed to the database. Their files remained in temporary object storage and were recovered automatically after service restoration.
- Seven metadata updates could not be recovered and were re-entered manually from audit logs.

Approximately 84% of active users encountered at least one error. Original media files already stored in the library were not corrupted or deleted.

## Timeline

All times are in EDT.

| Time | Event |
|---|---|
| 09:12 | Version `2026.07.14-1` was deployed, including a backfill to extract camera and lens metadata from older assets. |
| 09:18 | The backfill began processing approximately 4.8 million asset records. |
| 09:43 | Database disk utilization exceeded 85%. The warning alert was sent to an unmonitored email-only notification channel. |
| 10:11 | The nightly backup job began failing because the backup staging directory contained an incomplete snapshot from the previous night. |
| 10:27 | WAL archive uploads stopped succeeding. PostgreSQL retained WAL segments required by the failed archive process. |
| 10:46 | Database disk utilization exceeded 95%. No page was triggered because the paging threshold was configured only for the host filesystem, not the dedicated database volume. |
| 11:03 | The database volume reached 100% utilization. PostgreSQL began rejecting writes with `No space left on device`. |
| 11:05 | API error rates exceeded 60%, and the general availability alert paged the on-call engineer. |
| 11:12 | The on-call engineer identified PostgreSQL as unhealthy and attempted a restart. The restart failed because there was insufficient space for crash recovery. |
| 11:21 | The incident was escalated to SEV-1. Upload workers and background processors were stopped to prevent additional retries. |
| 11:34 | Engineers identified retained WAL and an abandoned 218 GB backup staging file as the primary disk consumers. |
| 11:46 | The failed backup process was confirmed inactive, and its incomplete staging file was removed. Approximately 218 GB was recovered. |
| 11:55 | PostgreSQL completed crash recovery and began accepting connections. |
| 12:03 | Read-only browsing was restored while upload and processing queues remained paused. |
| 12:18 | The database volume was expanded from 1.5 TB to 2 TB. |
| 12:27 | Upload workers were restarted with reduced concurrency. |
| 12:39 | Background jobs resumed, excluding the metadata backfill. |
| 12:52 | Error rates and queue latency returned to normal. The incident was declared resolved. |
| 14:16 | Recovery of orphaned upload objects completed. |
| 17:40 | Audit-log review and manual restoration of unrecoverable metadata changes completed. |

## Root Cause

The immediate cause was exhaustion of the PostgreSQL data volume.

The metadata backfill updated nearly every processed asset row, including rows whose extracted metadata had not changed. Those unnecessary updates produced sustained WAL growth approximately eight times higher than observed during staging tests.

Under normal conditions, archived WAL segments are removed after successful transfer to backup storage. However, the backup system had entered a failed state because an incomplete snapshot from the previous night occupied its staging directory. WAL archiving then failed continuously, causing PostgreSQL to retain every new WAL segment.

The combination of excessive WAL generation and failed archival consumed all remaining space on the database volume. PostgreSQL could no longer write WAL or temporary files and shut down to protect database consistency.

## Contributing Factors

- Staging tests used 120,000 assets and did not represent production data volume or metadata distribution.
- The backfill was not rate-limited and had no automatic pause threshold based on database disk utilization.
- The migration updated rows even when the resulting metadata was unchanged.
- Backup health was monitored through daily email reports rather than paging alerts.
- Disk alerts covered the root filesystem but not the separately mounted database volume.
- Capacity planning assumed normal application growth and did not include migration-generated WAL.
- The deployment runbook did not require confirming backup and WAL archival health before starting a write-heavy migration.
- Upload clients retried failed requests aggressively, increasing database connection pressure during recovery.
- The first restart attempt occurred before sufficient disk space had been recovered, extending the recovery period.

## Resolution and Recovery

Engineers stopped the metadata backfill and all nonessential background workers. After verifying that the backup process was no longer running and that the incomplete snapshot was not usable, they removed the abandoned staging file. This provided enough free space for PostgreSQL to complete crash recovery.

The database volume was then expanded by 500 GB. Services were restored gradually, beginning with read-only browsing, followed by uploads and background processing at reduced concurrency.

A reconciliation job compared temporary upload objects against committed asset records. It recovered twelve uploads whose files had reached object storage but whose database transactions had not completed. Audit logs were used to identify seven metadata changes requiring manual restoration.

## What Went Well

- PostgreSQL shut down cleanly enough to avoid database corruption.
- Original media was stored separately from the database and remained unaffected.
- Object-storage lifecycle rules retained temporary uploads long enough for reconciliation.
- Application audit logs provided sufficient information to identify incomplete metadata operations.
- Once disk exhaustion was identified, the team restored browsing within 58 minutes.

## What Went Poorly

- The first actionable alert came after customers were already affected.
- Backup failures did not page the on-call engineer.
- The migration lacked rate limits, progress controls, and a tested rollback procedure.
- Database-volume metrics existed but were not connected to the paging system.
- The initial restart attempt delayed diagnosis and could have increased recovery risk.
- The status page was not updated until 43 minutes after the outage began.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add paging alerts at 80%, 90%, and 95% utilization for every database volume. | Infrastructure | P0 | July 18, 2026 |
| Page on WAL archival failures lasting more than five minutes. | Infrastructure | P0 | July 18, 2026 |
| Add an automated migration pause when database free space falls below 20%. | Backend | P0 | July 22, 2026 |
| Change the metadata backfill to skip unchanged rows and commit in bounded batches. | Backend | P0 | July 22, 2026 |
| Add migration rate limits configurable without redeployment. | Backend | P1 | July 29, 2026 |
| Prevent backup jobs from starting when stale staging files are present, and clean them safely after validation. | Infrastructure | P1 | July 25, 2026 |
| Add backup freshness and restore-readiness checks to the operations dashboard. | Infrastructure | P1 | July 25, 2026 |
| Test write-heavy migrations against a production-scale anonymized dataset. | Engineering Productivity | P1 | August 5, 2026 |
| Require capacity estimates and backup-health verification in migration runbooks. | Engineering | P1 | July 24, 2026 |
| Add upload idempotency keys to prevent duplicate work during client retries. | API | P1 | August 12, 2026 |
| Add automatic reconciliation for object-storage uploads without committed asset records. | Media Platform | P1 | August 5, 2026 |
| Update the incident runbook to prohibit database restarts during disk exhaustion until recovery space is verified. | Reliability | P1 | July 21, 2026 |
| Automate status-page updates from SEV-1 incident creation. | Reliability | P2 | August 19, 2026 |
| Run a quarterly restore exercise covering database, metadata, and orphaned media recovery. | Reliability | P2 | September 15, 2026 |

## Lessons Learned

Background migrations must be treated as production workloads, not routine maintenance. Their write amplification, WAL generation, and storage requirements need to be measured at realistic scale before deployment.

Backup health is also part of service availability. A failed archive process can affect the live database even when no restore is being performed. Backup failures and database-capacity warnings now require the same operational urgency as elevated API error rates.