# Postmortem: Spring Bird Survey Submission Outage

**Incident date:** May 11, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Services affected:** Survey submission API, mobile synchronization, public observation map  
**Status:** Resolved

## Summary

During the first morning of the annual Spring Bird Count, participants were intermittently unable to submit bird observations through the website and mobile applications. A database connection surge caused the submission API to time out, while automatic retries from clients amplified the load.

The incident began at 06:18 UTC, shortly after participation increased across several European time zones. Full service was restored at 09:05 UTC after we disabled a newly introduced enrichment query, limited client retries, and increased database capacity.

No confirmed observations were lost. Some participants saw failed-submission messages even though their observations had been accepted, resulting in duplicate records that were later identified and merged.

## Customer Impact

Between 06:18 and 09:05 UTC:

- 38% of submission requests failed or timed out.
- Approximately 21,400 participants encountered at least one error.
- 184,000 observations were submitted later than intended.
- 12,760 duplicate observation records were created by repeated submissions.
- The public observation map lagged by up to three hours.
- Mobile users with offline storage could continue recording observations, but synchronization was delayed.
- Web users without offline storage had to retain their entries and resubmit them later.

Regional coordinators also experienced delays accessing participation totals and preliminary species counts.

## Timeline

All times are in UTC.

- **05:30** — The annual Spring Bird Count opens. Traffic begins increasing as expected.
- **06:12** — Submission API latency rises above the normal 300 ms baseline.
- **06:18** — Database connection usage reaches 95%. The first elevated error-rate alert fires.
- **06:23** — The on-call engineer acknowledges the alert and begins investigating API and database metrics.
- **06:29** — Submission failures exceed 20%. The incident is declared SEV-1.
- **06:37** — Engineers identify a high volume of long-running queries from the species-enrichment service.
- **06:45** — The team increases the database connection limit. Error rates briefly improve, then rise again as queued retries consume the additional capacity.
- **07:02** — Support publishes an incident notice advising mobile participants to continue recording observations offline.
- **07:16** — Analysis shows that version 4.18 of the submission API introduced a synchronous geographic enrichment query for every observation.
- **07:31** — The team disables geographic enrichment using a feature flag. Database query time drops, but retry traffic continues to overload the API.
- **07:48** — Server-side rate limits are applied to repeated submissions, and web-client retry intervals are increased.
- **08:10** — Submission success rate recovers to 92%. The backlog begins draining.
- **08:34** — Database replicas are temporarily scaled up to accelerate map and reporting updates.
- **09:05** — Submission success rate and latency return to normal. The incident is marked resolved.
- **11:40** — Delayed map updates finish processing.
- **16:20** — The duplicate-detection job identifies affected records; automated merging begins.
- **May 12, 14:00** — Duplicate cleanup completes, and regional coordinators confirm that summary counts are accurate.

## Root Cause

The immediate cause was a synchronous geographic enrichment query added in submission API version 4.18, deployed two days before the event. The query assigned each observation to a survey region by comparing its coordinates with regional boundary polygons.

The query performed adequately under ordinary traffic but did not use the intended spatial index because the application converted coordinates into a different spatial reference system inside the query. This forced the database to scan regional boundary data for every submitted observation.

At peak event traffic, these queries held database connections for several seconds. The submission API exhausted its connection pool, causing requests to time out. Both the web and mobile clients automatically retried timed-out requests with short, fixed delays, increasing traffic and preventing the service from recovering after additional database capacity was added.

## Contributing Factors

- Load testing used simplified regional boundary data and did not reproduce production query plans.
- Capacity estimates were based on the previous year’s average submission rate rather than expected peak traffic.
- The deployment was considered low risk because the enrichment result did not change the survey submission schema.
- The feature flag disabled only response decoration, not the underlying database query.
- Client retries did not use exponential backoff or randomized jitter.
- Submission requests lacked a consistently enforced idempotency key, allowing retries to create duplicates.
- Database alerts focused on CPU usage and connection count but did not alert on spatial-query duration.
- The event-day runbook did not include a tested procedure for disabling enrichment processing.

## Resolution and Recovery

Engineers disabled the enrichment query at the API layer, introduced temporary rate limits for repeated requests, and increased client retry intervals. Additional database capacity was added to process the accumulated map and reporting backlog.

After service recovery, we used participant, checklist, timestamp, location, and species data to identify duplicate observations. Records with identical idempotency characteristics were automatically merged, while ambiguous cases were reviewed by regional data coordinators.

## What Went Well

- Mobile offline storage preserved observations recorded during the outage.
- The database team quickly identified the abnormal query pattern.
- Feature flags allowed the enrichment workflow to be disabled without a full deployment.
- Support staff and regional coordinators communicated workarounds to participants within 45 minutes.
- Existing audit logs provided enough information to identify and merge duplicate records.

## What Did Not Go Well

- Pre-event testing did not reflect production data size or peak concurrency.
- Increasing database capacity initially worsened the retry surge instead of restoring service.
- Participants received ambiguous timeout messages and could not tell whether submissions had succeeded.
- The public status page depended on manual updates during the incident.
- Ownership of the enrichment service was unclear, delaying the decision to disable it.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Move geographic enrichment out of the synchronous submission path and process it asynchronously | Observations Platform | P0 | May 25, 2026 |
| Add and validate spatial indexes for all production boundary datasets | Data Infrastructure | P0 | May 18, 2026 |
| Enforce idempotency keys for all web and mobile submissions | API Team | P0 | June 8, 2026 |
| Implement exponential backoff and randomized jitter in client retries | Mobile and Web Teams | P0 | June 8, 2026 |
| Add peak-event load tests using production-scale boundary data and query plans | Reliability Engineering | P1 | June 15, 2026 |
| Alert on database query latency, pool wait time, and spatial sequential scans | Database Team | P1 | May 22, 2026 |
| Add a user-visible submission state distinguishing pending, accepted, and failed observations | Product Engineering | P1 | June 30, 2026 |
| Test all event-critical feature flags and document their complete effects | Release Engineering | P1 | June 1, 2026 |
| Update the event-day runbook with retry-storm and enrichment-disable procedures | Site Reliability Engineering | P1 | May 20, 2026 |
| Automate status-page updates from incident-management events | Developer Experience | P2 | July 15, 2026 |
| Conduct a peak-capacity review four weeks before every scheduled global survey | Engineering Management | P1 | Ongoing |

## Lessons Learned

Features outside the apparent critical path can still become critical when they share synchronous infrastructure with submissions. Event readiness must validate production-scale data, real query plans, client retry behavior, and failure modes—not only nominal request throughput.

We also learned that offline capture protects observations but does not eliminate participant uncertainty. Clear submission states and idempotent APIs are necessary to make recovery safe and understandable during degraded service.