# Postmortem: Bird Survey Submission Outage

Date: 2026-07-08  
Status: Final  
Severity: SEV-2  
Duration: 1 hour 42 minutes  
Services affected: Survey API, mobile sync, public dashboard ingestion

## Summary

On July 8, 2026, participants in the citizen-science bird survey were unable to submit new observations or sync offline checklists. The outage occurred during the peak morning survey window and affected both the mobile app and web submission form.

The immediate cause was database connection exhaustion after a scheduled dashboard backfill job opened significantly more concurrent read connections than expected. This prevented the Survey API from acquiring write connections for incoming submissions.

## Customer Impact

From 06:18 to 08:00 UTC:

- 68% of submission attempts failed with HTTP 503 or timed out.
- Approximately 14,200 checklists were delayed.
- 1,930 users saw repeated sync failures in the mobile app.
- No submitted observations were permanently lost.
- Public species-count dashboards were stale for up to 2 hours.
- Regional coordinators were unable to monitor live participation during the morning survey window.

Offline mobile submissions were queued locally and synced after service recovery.

## Timeline

All times are UTC.

| Time | Event |
|---|---|
| 05:55 | Scheduled dashboard backfill job started for the spring migration dataset. |
| 06:12 | Database read connection count began rising above normal baseline. |
| 06:18 | Survey API error rate exceeded 20%; users began seeing failed submissions. |
| 06:22 | On-call engineer was paged by API availability alert. |
| 06:28 | Initial investigation showed elevated API latency and database connection pool saturation. |
| 06:36 | Backfill job was identified as the largest source of active database sessions. |
| 06:41 | Backfill job was cancelled. Existing database connections did not drain immediately. |
| 06:50 | API instances were restarted to clear blocked connection pools. |
| 07:05 | Submission success rate began recovering, but mobile sync backlog caused elevated load. |
| 07:22 | Rate limits were temporarily increased for authenticated mobile sync traffic. |
| 07:44 | Submission success rate returned above 99%. |
| 08:00 | Incident was declared resolved. |
| 09:30 | Delayed dashboard aggregates were fully caught up. |

## Root Cause

The outage was caused by an analytics backfill job that bypassed the standard database connection pool configuration. The job created one database connection per worker shard, and the shard count was automatically increased after a recent data volume change.

This consumed nearly all available database connections. As a result, the Survey API could not reliably acquire connections to write new observations, update checklist state, or acknowledge mobile sync requests.

## Contributing Factors

- The backfill job used a separate database client configuration from production services.
- The job ran during a known high-traffic survey window.
- Alerts detected API failure rate but did not directly alert on database connection saturation.
- The mobile app retried failed syncs aggressively, increasing load during recovery.
- Dashboard backfills and live survey submissions shared the same primary database.

## Resolution

The team cancelled the backfill job, restarted affected API instances, and temporarily adjusted sync rate limits to allow queued mobile submissions to drain in a controlled way.

No observation data was lost. Mobile clients retained unsynced checklists locally and uploaded them after the API recovered.

## What Went Well

- On-call was paged within 4 minutes of user-visible errors.
- Offline mobile storage prevented permanent data loss.
- The failing backfill job was identified and stopped quickly.
- Regional coordinators received an incident notice before the survey window ended.

## What Went Wrong

- The backfill job was allowed to run during peak collection hours.
- Database connection limits were not enforced consistently across services.
- Retry behavior amplified load during partial recovery.
- Public dashboards showed stale data without a clear degraded-state message.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---|---|
| Move analytics backfills to a read replica with separate connection limits. | Data Platform | 2026-07-22 | Open |
| Enforce shared database connection pool defaults across all services and jobs. | Backend Platform | 2026-07-19 | Open |
| Block non-urgent backfills during regional peak survey windows. | Data Engineering | 2026-07-15 | Open |
| Add alerts for database connection saturation and long-running job connection usage. | SRE | 2026-07-16 | Open |
| Reduce mobile sync retry aggressiveness with exponential backoff and jitter. | Mobile | 2026-07-26 | Open |
| Add dashboard degraded-state messaging when ingestion is delayed. | Web | 2026-07-24 | Open |
| Run a recovery drill for queued mobile submissions at 5x normal morning traffic. | SRE | 2026-08-02 | Open |

## Prevention

We will isolate analytics workloads from live survey writes, enforce consistent connection limits, and prevent heavy background jobs from running during expected collection peaks. We will also improve client retry behavior so recovery traffic does not extend future incidents.