# Postmortem: Bird Survey Submission Outage — May 17, 2026

**Incident date:** May 17, 2026  
**Duration:** 2 hours 41 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Authors:** Survey Platform Engineering  
**Last updated:** May 22, 2026

## Summary

On May 17, the first morning of the annual Spring Bird Count, participants were unable to submit new survey checklists or edit existing drafts. The outage began at 06:12 UTC, shortly after participation increased across Europe, and lasted until 08:53 UTC.

The submission API returned errors for 68% of write requests during the incident. Read-only features—including species guides, public maps, and previously submitted observations—remained available, although some pages loaded slowly.

The immediate cause was database connection exhaustion in the survey submission service. A deployment made the previous day introduced an unbounded retry path for geospatial validation queries. When database latency increased under event traffic, requests retried without releasing their existing connections. This created a feedback loop: retries consumed the remaining connection capacity, which increased latency and triggered more retries.

We restored service by disabling synchronous geospatial validation, restarting the affected application instances, and temporarily increasing database connection capacity. No confirmed survey records were lost. Some participants had to re-enter observations that were held only in unsaved browser sessions.

## Customer Impact

Approximately 41,700 participants used the platform during the incident window.

- 28,436 survey submission attempts failed.
- 11,902 unique participants encountered at least one error.
- 3,184 draft-edit requests failed.
- 1,276 mobile uploads were delayed and later synchronized automatically.
- 642 support requests were received during the first six hours.
- Public maps and species pages experienced elevated latency but remained operational.

Participants saw one of three symptoms:

- A generic “Unable to submit survey” error after pressing **Submit**.
- A loading indicator that timed out after 30 seconds.
- A validation message stating that the survey location could not be verified.

Submitted records are written transactionally, so the incident did not create partial or corrupted surveys. We reconciled client-generated submission IDs against database records and found 317 apparent duplicates caused by manual resubmission. These duplicates were quarantined for review rather than automatically deleted.

Observations saved locally by the mobile applications synchronized after recovery. The web application did not yet support durable offline drafts, so observations entered into an unsaved browser form may have been lost if the page was refreshed or closed.

## Detection

The incident was first reported by participants through the support queue at 06:18 UTC. Engineering was paged at 06:27 UTC by an alert on overall API error rate.

Our existing alerting delayed detection because it evaluated errors across all API endpoints. Healthy read traffic diluted the submission failures, and the aggregate error rate did not cross the paging threshold until 15 minutes after the outage began.

## Timeline

All times are in UTC on May 17, 2026.

| Time | Event |
|---|---|
| 06:12 | Database latency begins increasing as Spring Bird Count traffic ramps up. Submission error rates rise above 20%. |
| 06:18 | Support receives the first participant reports of failed submissions. |
| 06:23 | Database connection utilization reaches 95%. |
| 06:27 | The on-call engineer is paged for elevated aggregate API errors. |
| 06:31 | Incident declared SEV-1. Engineering, database operations, support, and event operations join the incident channel. |
| 06:38 | The team identifies connection-pool timeouts in the survey submission service. |
| 06:44 | Application instances are restarted in batches. Error rates briefly decline, then return as traffic reconnects. |
| 06:51 | A temporary database connection-limit increase is approved and applied. Successful submissions rise from 32% to 54%, but latency remains high. |
| 07:03 | Engineers correlate the failures with geospatial validation queries introduced in release `survey-api-2026.05.16.3`. |
| 07:17 | Load testing in staging reproduces connection leakage when validation queries time out and retry. |
| 07:29 | The team decides to disable synchronous geospatial validation through a feature flag. |
| 07:36 | The feature flag is enabled for 10% of production traffic. Submission success improves without an increase in invalid location data. |
| 07:43 | Synchronous validation is disabled for all survey submissions. Database connection utilization begins falling. |
| 07:51 | Submission success exceeds 95%. Mobile synchronization backlog starts draining. |
| 08:14 | Database latency returns to normal. Duplicate-submission reconciliation begins. |
| 08:37 | Mobile synchronization backlog is cleared. |
| 08:53 | Error rates and latency remain within normal ranges for 30 minutes. Incident marked resolved. |
| 10:20 | Support publishes a participant-facing incident summary and recovery guidance. |
| 15:40 | Reconciliation confirms no loss of successfully acknowledged submissions. |

## Root Cause

The root cause was an application defect in a new geospatial validation workflow deployed on May 16.

Before accepting a survey, the submission service checks whether its coordinates fall within the selected survey region. The new workflow retried this database query up to three times when it encountered a timeout. However, the retry logic executed inside the scope of the original database transaction. Each retry requested another connection while the previous connection remained checked out until the request completed.

Under normal traffic, the validation query completed quickly and the defect had little visible effect. During the Spring Bird Count, concurrent submissions increased approximately sixfold. A subset of geospatial queries became slow because the new query plan performed poorly for large coastal survey regions. Those queries timed out and entered the defective retry path.

The resulting failure loop was:

1. Increased event traffic caused geospatial validation queries to slow down.
2. Slow queries timed out while retaining their database connections.
3. Retries requested additional connections from the same pool.
4. Connection availability fell, increasing wait times for unrelated submissions.
5. More requests timed out and retried, exhausting both application and database connection capacity.

Restarting application instances released connections but did not address the trigger, so exhaustion returned as soon as traffic resumed.

## Contributing Factors

- The retry implementation was tested for successful recovery but not for connection ownership during repeated timeouts.
- Performance tests used uniformly distributed coordinates and did not include large or geometrically complex survey regions.
- Pre-event load testing reached only 55% of the traffic observed during the incident.
- The database connection pool allowed requests to wait longer than the API gateway timeout, consuming resources after clients had already received errors.
- Synchronous geospatial validation was on the critical submission path even though the same check could be performed asynchronously.
- The deployment occurred less than 24 hours before the annual event without an event-specific change freeze.
- Alerts measured aggregate API availability instead of submission success separately.
- The web client did not preserve unsaved survey drafts across refreshes or browser closures.
- The participant-facing status page required a manual update, delaying public communication.

## Resolution and Recovery

We restored service by:

1. Disabling synchronous geospatial validation with an existing feature flag.
2. Restarting application instances to release stranded connections.
3. Temporarily increasing the database connection limit to support recovery traffic.
4. Draining queued mobile submissions at a controlled rate.
5. Reconciling client-generated submission IDs to identify duplicate attempts.

Location validation now runs asynchronously after a survey is accepted. Surveys that fail validation are placed in a review queue rather than rejected during submission.

The temporary database limit increase was reverted after traffic normalized. The defective release was removed from production, and the retry path was replaced with a bounded operation that explicitly releases its connection before retrying.

## What Went Well

- Transactional writes prevented partial or corrupted survey records.
- Client-generated submission IDs enabled reliable reconciliation of repeated attempts.
- Mobile applications retained offline observations and synchronized them after recovery.
- The geospatial validation feature flag allowed the team to remove the failing dependency without a full deployment.
- Support and event operations quickly consolidated participant reports and published recovery instructions.

## What Went Poorly

- Submission failures began 15 minutes before the first engineering page.
- Restarting instances provided only temporary relief and increased pressure on the database during reconnection.
- The status page was not updated until more than an hour after the incident began.
- The web application offered no durable protection for unsaved observations.
- Event traffic assumptions were based on the previous year and did not account for growth from new regional partners.
- Runbooks covered general connection exhaustion but not transaction-scoped retry behavior.

## Action Items

| Action | Owner | Priority | Due date | Status |
|---|---|---:|---|---|
| Remove nested database acquisition from all transaction retry paths and add regression tests for connection release. | Survey API Team | P0 | May 24 | In progress |
| Add per-endpoint alerts for submission success rate, latency, and connection-pool saturation. | Reliability Engineering | P0 | May 23 | In progress |
| Keep geospatial validation asynchronous and add a review workflow for invalid locations. | Data Quality Team | P0 | May 31 | In progress |
| Add load-test scenarios for complex regions, retry storms, and database degradation. | Performance Engineering | P1 | June 7 | Planned |
| Increase pre-event capacity testing to 150% of forecast peak traffic. | Reliability Engineering | P1 | June 14 | Planned |
| Align application connection wait time with the API gateway timeout. | Database Operations | P1 | May 28 | Planned |
| Introduce a seven-day production change freeze before major annual survey events. | Engineering Leadership | P1 | June 1 | Planned |
| Implement durable browser drafts using local storage with clear privacy and expiry controls. | Web Team | P1 | June 21 | Planned |
| Automatically update the public status page from incident-management events. | Platform Team | P2 | June 30 | Planned |
| Add retry and connection-exhaustion guidance to the submission-service runbook. | Survey API Team | P2 | May 27 | Planned |
| Review and merge the 317 quarantined duplicate submissions with regional coordinators. | Data Operations | P1 | May 29 | In progress |
| Publish participant guidance for recovering or re-entering unsaved observations. | Support | P2 | May 24 | Complete |

## Prevention

Before the next major survey event, we will conduct a full-scale load exercise that includes degraded database behavior and delayed geospatial queries. Readiness approval will require evidence that submission traffic remains available when optional validation systems fail.

We will also treat survey acceptance as the smallest critical path: once identity, required fields, and idempotency checks pass, optional enrichment and data-quality processing will occur asynchronously. This reduces the number of dependencies capable of preventing participants from recording time-sensitive observations.