# Postmortem: Bird Survey Submission Outage

**Incident date:** 2026-05-10  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

During peak migration survey hours, participants in the citizen-science bird survey were unable to submit observations through the web and mobile apps. The API returned elevated `503` and `504` responses after the primary database connection pool was exhausted.

The outage was triggered by a recent species-autocomplete change that generated excessive database queries under high traffic. The issue was mitigated by disabling the autocomplete endpoint feature flag, restarting saturated API workers, and increasing database connection pool headroom.

## Customer Impact

- Approximately 38% of observation submissions failed during the incident window.
- 12,400 participants were affected.
- 47,000 observation checklists were delayed or required retry.
- Mobile users experienced repeated submission failures and stale sync states.
- Research partners received incomplete near-real-time migration dashboards until backfill completed.

No submitted observations were permanently lost. Offline mobile records were resubmitted successfully after service recovery.

## Timeline

All times are Eastern Time.

| Time | Event |
|---|---|
| 07:42 | New species-autocomplete ranking model deployed to production. |
| 08:03 | Traffic increased as scheduled survey reminders were sent. |
| 08:11 | API latency exceeded 2 seconds at p95. |
| 08:18 | First alerts fired for elevated `POST /observations` error rate. |
| 08:22 | On-call engineer declared SEV-1 and opened incident channel. |
| 08:31 | Database metrics showed connection pool exhaustion and slow query growth. |
| 08:44 | Team identified autocomplete endpoint as the highest query source. |
| 08:52 | Autocomplete feature flag disabled globally. |
| 09:05 | API workers restarted to clear blocked request queues. |
| 09:21 | Observation submissions returned to normal success rates. |
| 09:47 | Backfill job started for delayed mobile sync submissions. |
| 10:20 | Dashboards confirmed all queued observations processed. |
| 10:28 | Incident resolved. |

## Root Cause

The root cause was an inefficient query pattern introduced in the new species-autocomplete ranking model. For each typed character, the web and mobile clients requested updated suggestions. The backend performed multiple uncached database lookups per request to combine species names, regional likelihood, recent sightings, and user history.

Under normal load this passed staging tests, but during the morning survey peak the request volume amplified sharply. The database connection pool became saturated, causing unrelated observation submission requests to queue behind autocomplete queries. Once API workers were blocked waiting for database connections, submission requests began timing out.

## Contributing Factors

- Staging load tests did not model peak survey reminder traffic.
- Autocomplete requests were not rate-limited or debounced consistently across clients.
- The endpoint lacked a circuit breaker and shared the same database pool as submission writes.
- Alerts detected the symptom, failed submissions, before identifying database pool saturation.
- The deployment happened during a known high-traffic survey window.

## Resolution

The incident was resolved by disabling the autocomplete feature flag, restarting saturated API workers, and temporarily increasing the database connection pool size. Delayed mobile submissions were replayed from local queues and server-side retry buffers.

## What Went Well

- Feature flags allowed the problematic endpoint to be disabled without a rollback.
- Offline mobile storage prevented permanent observation loss.
- Incident response roles were assigned quickly.
- Backfill tooling processed delayed submissions without manual data repair.

## What Went Poorly

- The deploy was allowed during a peak participation window.
- Load testing missed a realistic client behavior pattern.
- Submission writes and autocomplete reads competed for the same database resources.
- Status page updates lagged behind the internal incident timeline.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---|---|
| Add production-like load tests for survey reminder traffic. | Platform | 2026-05-24 | Open |
| Move autocomplete reads to cached search index. | Data Products | 2026-06-07 | Open |
| Add client-side debounce and minimum character threshold. | Mobile/Web | 2026-05-21 | Open |
| Separate database pools for critical writes and optional reads. | Infrastructure | 2026-06-14 | Open |
| Block non-critical deploys during scheduled survey peaks. | Release Engineering | 2026-05-17 | Open |
| Add alerts for database pool saturation by endpoint. | SRE | 2026-05-24 | Open |
| Update incident runbook with feature-flag rollback steps. | SRE | 2026-05-18 | Open |
| Improve status page update ownership during SEV-1 incidents. | Support | 2026-05-20 | Open |