# Postmortem: Spring Bird Survey Submission Outage

**Incident date:** May 11, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Affected services:** Web survey submission, mobile synchronization, public sightings map  
**Status:** Resolved

## Summary

During the annual Spring Bird Count, participants were unable to submit new observations through the website or synchronize surveys from the mobile application. The public sightings map also stopped updating.

The incident began when peak submission traffic exhausted the database connection pool used by the survey ingestion service. Automatic retries amplified the load, preventing the service from recovering after traffic declined. We restored service by disabling application-level retries, increasing connection capacity, and gradually draining the submission backlog.

No confirmed observations were permanently lost. Some participants submitted duplicate surveys after retrying, and a small number of mobile users had to reopen the application to complete synchronization.

## Customer Impact

Between 08:42 and 11:29 EDT:

- Approximately 18,400 participants encountered submission or synchronization failures.
- 31,762 survey submissions were delayed.
- 4,286 users saw a generic “Unable to submit survey” error.
- The public sightings map displayed data up to 96 minutes behind real time.
- 1,143 duplicate surveys were created and later reconciled.
- Support received 687 emails and 214 social-media reports.

Mobile observations saved locally remained on participants’ devices until synchronization succeeded. Server-side audit logs and client-generated submission identifiers allowed us to verify that no accepted observation records were lost.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 08:42 | Database connection utilization reaches 100%. Submission latency begins increasing. |
| 08:45 | First automated alert fires for elevated API error rate. |
| 08:49 | On-call engineer acknowledges the alert and begins investigating the survey ingestion service. |
| 08:55 | Submission success rate falls below 40%. Incident is declared SEV-1. |
| 09:03 | Team identifies database connection exhaustion but initially suspects a slow reporting query. |
| 09:14 | Reporting jobs are paused. Database load decreases briefly, but submission failures continue. |
| 09:27 | Engineers discover that failed requests are being retried by the mobile client, API gateway, and ingestion worker. |
| 09:36 | Public status page is updated. Support is instructed to tell participants not to delete unsynchronized surveys. |
| 09:48 | API gateway retries are disabled. Error rate stabilizes but remains elevated. |
| 10:02 | Application-level retries are disabled through a configuration change. |
| 10:15 | Database connection pool limit is increased from 120 to 220 after capacity checks. |
| 10:29 | Submission success rate recovers above 90%. Backlog processing begins. |
| 10:51 | Mobile synchronization returns to normal. |
| 11:07 | Public map updates resume. |
| 11:29 | Backlog is cleared, metrics remain healthy, and the incident is marked resolved. |
| 14:10 | Duplicate-survey reconciliation job begins. |
| 17:45 | Duplicate reconciliation completes. |

## Root Cause

The immediate cause was exhaustion of the PostgreSQL connection pool used by the survey ingestion service.

A deployment three days before the incident changed submission processing so that each observation’s media metadata was validated within the same database transaction as the survey record. For surveys containing photographs or audio clips, this increased the median transaction duration from 180 milliseconds to 1.4 seconds.

Normal weekday traffic did not expose the problem. During the Spring Bird Count peak, concurrent submissions increased to roughly six times the usual maximum. Long-running transactions occupied all available database connections, causing requests to time out while waiting for a connection.

The failure was amplified by three independent retry mechanisms:

1. The mobile client retried failed synchronization requests.
2. The API gateway retried requests that timed out before receiving response headers.
3. The ingestion worker retried timed-out transactions immediately.

Because retries lacked exponential backoff and shared idempotency guarantees, each timeout generated additional database work. This created a retry storm that continued after the initial traffic peak and produced duplicate submissions.

## Contributing Factors

- Load tests modeled typical daily traffic rather than the annual event’s expected peak.
- The deployment’s transaction-duration increase was visible in metrics but did not cross an alert threshold.
- Media validation involved a network call while a database transaction remained open.
- Retry behavior was configured independently across three system layers.
- The connection-pool alert warned only after utilization exceeded 95% for five minutes.
- The submission endpoint did not consistently enforce client-generated idempotency keys.
- The incident runbook did not document how to disable retries without a deployment.
- The public status page required a separate login that slowed the first customer communication.

## Resolution and Recovery

We disabled retries at the API gateway and ingestion service, then increased the database connection-pool limit after confirming sufficient database headroom. This allowed new requests to complete and the queued submissions to drain without generating additional retry traffic.

After service recovery, we used client-generated submission identifiers, timestamps, participant IDs, and observation contents to identify duplicate surveys. Ambiguous cases were reviewed manually before records were merged.

## What Went Well

- Client applications retained unsynchronized observations locally.
- Audit logs contained enough information to confirm data integrity.
- Database and API telemetry made connection exhaustion visible.
- Engineering, community support, and communications teams coordinated through a single incident channel.
- Retry settings could be changed at runtime once the relevant configuration was identified.

## What Went Poorly

- Alerts detected the incident after participants were already experiencing failures.
- Initial investigation focused on reporting queries, delaying identification of the retry storm.
- Multiple teams owned different retry layers, with no documented end-to-end retry policy.
- The generic client error message encouraged some participants to resubmit manually.
- Event-specific capacity planning was based on the previous year’s average throughput rather than short-duration peaks.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Move media validation outside the database transaction. | Survey Platform | P0 | May 15, 2026 |
| Enforce idempotency keys for all survey submissions and synchronization requests. | API Team | P0 | May 22, 2026 |
| Implement exponential backoff with jitter and retry budgets across clients, gateway, and workers. | Reliability Engineering | P0 | May 29, 2026 |
| Add load tests modeling ten times the expected annual-event peak, including media attachments. | Performance Engineering | P1 | June 5, 2026 |
| Alert on connection wait time, transaction duration, and pool saturation before exhaustion. | Observability Team | P1 | May 20, 2026 |
| Add a circuit breaker that pauses backlog processing when interactive submissions degrade. | Survey Platform | P1 | June 12, 2026 |
| Create a single documented retry policy and assign an owner for end-to-end enforcement. | Architecture Group | P1 | May 27, 2026 |
| Update mobile messaging to confirm that failed surveys remain stored locally. | Mobile Team | P1 | June 3, 2026 |
| Add retry controls and database-pool mitigation steps to the incident runbook. | Reliability Engineering | P1 | May 18, 2026 |
| Run a capacity review and failure exercise two weeks before every scheduled survey event. | Program Operations | P2 | July 1, 2026 |
| Provide incident responders with direct, tested access to the public status page. | Communications | P2 | May 19, 2026 |

## Prevention

Before the next large survey event, we will validate the complete submission path under projected peak traffic, including slow media processing, partial dependency failures, and client retries. Readiness approval will require evidence that submissions remain idempotent, database transactions contain no network calls, and overload controls preserve interactive traffic while safely delaying background work.