# Postmortem: Photo Library Upload and Browsing Outage

**Incident date:** July 28, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Services affected:** Web application, mobile uploads, thumbnail generation, metadata search  
**Status:** Resolved

## Summary

On July 28, the self-hosted photo library became unavailable after the PostgreSQL volume reached 100% capacity. A scheduled metadata reindex generated significantly more temporary data than expected while a backup job simultaneously wrote a full database snapshot to the same disk.

Once the volume filled, PostgreSQL began rejecting writes and entered crash recovery after a restart. The application continued accepting upload requests briefly, but could not persist asset records. Browsing, search, album updates, and background processing subsequently failed because database connections timed out.

Service was restored by stopping the reindex and backup jobs, expanding the database volume, removing incomplete temporary files, and allowing PostgreSQL recovery to complete.

## Customer Impact

During the incident:

- The web and mobile applications returned intermittent `500` and `503` errors.
- Existing photos and albums could not be loaded reliably.
- Search and face-recognition results were unavailable.
- New uploads failed or remained indefinitely in a “processing” state.
- Album edits, favorites, and metadata changes could not be saved.
- Background thumbnail and transcoding jobs stopped processing.

A total of 6,482 upload attempts failed. Of those, 6,109 were retried automatically after recovery. The remaining 373 required clients to retry manually because their temporary upload files had already expired.

No original photo or video files were lost. Twenty-seven album and favorite updates acknowledged by the API during the first four minutes of the incident were not committed and had to be repeated.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 01:00 | Weekly metadata reindex begins. |
| 01:05 | Nightly full database backup begins on the same storage volume. |
| 01:31 | Database volume utilization exceeds 90%. No alert is sent because the alert threshold is configured only for the host filesystem. |
| 01:42 | PostgreSQL temporary files consume the remaining free space. The volume reaches 100% capacity. |
| 01:43 | PostgreSQL begins returning write errors. Application error rates increase. |
| 01:45 | Upload workers start retrying failed database transactions, increasing connection usage and log volume. |
| 01:47 | The database health check fails. The container runtime restarts PostgreSQL. |
| 01:49 | PostgreSQL enters crash recovery but cannot complete because insufficient free space remains. |
| 01:52 | The web application becomes broadly unavailable as its database connection pool is exhausted. |
| 02:03 | The on-call engineer receives an application availability alert. |
| 02:11 | Initial investigation identifies database connection failures and repeated PostgreSQL restarts. |
| 02:19 | The database volume is confirmed to be full. |
| 02:24 | The metadata reindex and backup jobs are stopped. |
| 02:31 | Incomplete backup and temporary reindex files are removed, restoring 18 GB of free space. |
| 02:36 | PostgreSQL starts successfully and begins crash recovery. |
| 02:51 | Crash recovery completes. Read operations begin succeeding. |
| 03:02 | Application and worker containers are restarted to clear exhausted connection pools. |
| 03:09 | Browsing and album operations return to normal. |
| 03:14 | Upload processing resumes with reduced worker concurrency. |
| 03:27 | Failed upload jobs begin replaying from the queue. |
| 03:48 | Error rates, queue depth, and database latency return to normal. Incident resolved. |

## Root Cause

The immediate cause was exhaustion of the PostgreSQL storage volume.

The weekly metadata reindex created large temporary sort files because the operation exceeded PostgreSQL’s available working memory. At the same time, the nightly backup process wrote an uncompressed full database snapshot to the same volume. Together, these workloads consumed approximately 96 GB of previously free space in under 45 minutes.

The database restart policy then worsened the outage. When the database health check failed, the container runtime restarted PostgreSQL. PostgreSQL required additional disk space to complete crash recovery, but the volume was still full, resulting in a restart loop.

## Contributing Factors

- The reindex and backup schedules overlapped.
- Backups were written to the database volume before being transferred to object storage.
- Capacity alerts monitored the host filesystem but not the mounted database volume.
- No automated disk-space check ran before either maintenance job.
- PostgreSQL temporary-file usage had no configured limit.
- Upload workers retried database failures without exponential backoff, increasing load during recovery.
- The application reported success for a small number of mutations before confirming transaction commit.
- The operational runbook did not document recovery from a full PostgreSQL volume.
- Recent library growth had reduced free-space headroom from 38% to 14%, but capacity forecasts had not been updated.

## Resolution and Recovery

Responders stopped both maintenance jobs and removed incomplete backup and temporary files. The database volume was expanded from 500 GB to 750 GB, after which PostgreSQL completed crash recovery successfully.

Application and worker containers were restarted to clear stale database connections. Upload worker concurrency was temporarily reduced from 16 to 4 while queued jobs were replayed. Database consistency checks and a comparison against the object store confirmed that no original media files were lost.

## What Went Well

- Existing media files were stored separately from the database and were unaffected.
- The upload queue retained most failed jobs for automatic replay.
- Database checksums and crash recovery completed without corruption.
- Once the full volume was identified, responders restored enough space for recovery within 17 minutes.
- Post-recovery consistency checks were already available as maintenance scripts.

## What Went Poorly

- Monitoring detected application symptoms rather than the underlying storage condition.
- The database restart loop obscured the original disk-full error.
- Maintenance jobs shared storage and had no coordination mechanism.
- Upload retries amplified database pressure during the incident.
- Some API responses were returned before transaction durability was confirmed.
- The lack of a documented recovery procedure increased diagnosis time.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add alerts for database-volume usage at 70%, 80%, and 90% | Platform | P0 | August 2 |
| Move database backups to a dedicated staging volume | Platform | P0 | August 4 |
| Prevent backup and reindex jobs from running concurrently | Application | P0 | August 4 |
| Add preflight free-space checks to maintenance jobs | Application | P0 | August 7 |
| Configure a PostgreSQL temporary-file size limit | Database | P1 | August 7 |
| Add exponential backoff and jitter to worker database retries | Application | P1 | August 11 |
| Require confirmed transaction commit before mutation APIs return success | Application | P1 | August 14 |
| Add disk-capacity projections to the weekly operations report | Platform | P1 | August 14 |
| Test recovery from a full database volume in staging | Reliability | P1 | August 18 |
| Document the disk-full recovery procedure in the on-call runbook | Reliability | P1 | August 5 |
| Add an alert for repeated database container restarts | Platform | P2 | August 11 |
| Review retention and compression settings for database backups | Database | P2 | August 18 |

## Prevention

The primary preventive changes are isolating backup storage, eliminating overlapping maintenance jobs, and monitoring the database volume directly. Together, these measures address both the storage exhaustion and the delay in detecting it.

We will also run quarterly failure exercises covering full disks, database recovery, and upload-queue replay to verify that the service fails safely and that recovery procedures remain current.