# Postmortem: Survey Submission Outage During Spring Migration Count

**Incident date:** May 11, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** Web survey submission, mobile synchronization, reviewer dashboard

## Summary

On May 11, 2026, participants in the Spring Migration Count were unable to submit bird observations through the website or synchronize completed checklists from the mobile app. The incident occurred during the survey’s busiest morning and affected approximately 38% of submission attempts made during the outage.

A database migration deployed earlier that morning added a uniqueness constraint intended to prevent duplicate observations. The constraint did not account for legitimate cases where multiple participants submitted observations from the same survey location and time window. These rejected writes triggered aggressive client retries, increasing database load and exhausting the API’s connection pool.

We restored service by disabling automatic retries, removing the invalid constraint, and increasing API capacity. Mobile observations remained stored locally and synchronized after recovery. We found no evidence of permanent data loss, although some web participants had to re-enter observations.

## Customer Impact

Between 06:18 and 09:05 Eastern Time:

- 4,812 of 12,641 submission attempts failed.
- Approximately 2,900 participants encountered at least one error.
- 1,146 mobile checklists were delayed but synchronized after service recovery.
- An estimated 620 web checklists required manual re-entry because the web form did not preserve all fields after a failed submission.
- Regional coordinators could not review incoming observations reliably.
- Public species-count dashboards displayed incomplete totals for approximately four hours.

Participants saw one of the following messages:

- “Unable to submit checklist. Please try again.”
- “Observation already exists.”
- “Synchronization delayed.”

No existing observations were deleted or modified. A reconciliation completed later that day confirmed that all mobile checklists retained on devices were eventually synchronized.

## Detection

The incident was first detected through participant support reports at 06:24. Automated monitoring alerted at 06:31 when API error rates exceeded the configured threshold.

Detection was delayed because the first failures returned HTTP `409 Conflict`, which was classified as a client error and excluded from the primary availability alert. The alert fired only after retry traffic caused database connection failures and HTTP `503` responses.

## Timeline

All times are in Eastern Time on May 11, 2026.

| Time | Event |
|---|---|
| 05:42 | Database migration begins during the scheduled deployment window. |
| 05:49 | Migration completes successfully and adds a uniqueness constraint on survey site, observation time window, and species. |
| 06:00 | Spring Migration Count opens across participating eastern regions. |
| 06:18 | First legitimate submission is rejected by the new constraint. |
| 06:22 | Mobile and web clients begin retrying rejected submissions. |
| 06:24 | Support receives the first participant report of failed submissions. |
| 06:31 | API availability alert fires after database connection timeouts produce elevated `503` responses. |
| 06:35 | On-call engineer declares a SEV-1 incident and pauses unrelated deployments. |
| 06:43 | Initial investigation identifies database connection-pool exhaustion. API instances are scaled from 12 to 24. |
| 06:51 | Error rates briefly decline, then rise as queued mobile clients retry synchronization. |
| 07:04 | Engineers identify a high rate of uniqueness-constraint violations in database logs. |
| 07:18 | Incident team confirms the constraint rejects valid observations from different participants at shared survey sites. |
| 07:31 | Automatic retries are disabled through a remote client configuration. |
| 07:46 | Database load begins to decline, but submissions continue to fail because the invalid constraint remains active. |
| 08:02 | The team prepares and validates a migration to remove the constraint. |
| 08:27 | Corrective migration begins. |
| 08:36 | Invalid constraint is removed. New submissions begin succeeding. |
| 08:44 | Error rate returns below 1%; mobile synchronization backlog starts draining. |
| 09:05 | Service is declared restored. |
| 10:12 | Mobile synchronization backlog is fully processed. |
| 13:40 | Data reconciliation confirms no server-side observation loss and identifies incomplete web drafts requiring participant outreach. |
| 16:20 | Public dashboards are fully refreshed with reconciled counts. |

## Root Cause

The primary cause was an incorrectly designed database uniqueness constraint introduced to prevent duplicate observation records.

The constraint treated the following combination as globally unique:

- Survey site
- Observation time window
- Species

This model assumed that only one record for a species should exist at a site during a given time window. That assumption was incorrect. The survey permits multiple independent participants and teams to submit observations from the same site, and each submission must be retained for later validation.

The correct uniqueness boundary is the participant’s checklist and its client-generated observation identifier, not the physical site and time window.

When valid inserts violated the constraint, the API returned `409 Conflict`. Both clients treated every `409` as a transient synchronization conflict and retried with exponential backoff. Because thousands of participants were submitting observations during the morning peak, retries amplified traffic until the database connection pool was exhausted. This caused otherwise valid reads and writes to fail with `503 Service Unavailable`.

## Contributing Factors

- The migration was tested with synthetic data containing one participant per survey site.
- The schema review focused on database performance and did not include a survey-domain expert.
- Load tests modeled normal write volume but did not model synchronized retries after rejected writes.
- Client retry logic did not distinguish permanent validation conflicts from transient network failures.
- Availability monitoring excluded `409` responses, delaying automated detection.
- The web form cleared some unsaved fields after a failed submission.
- The migration was deployed shortly before the survey’s highest-traffic period.
- The rollback procedure required a new migration rather than allowing the constraint to be disabled quickly.

## Resolution and Recovery

The incident team took the following steps:

1. Doubled API capacity to reduce immediate connection pressure.
2. Disabled automatic retries using remote client configuration.
3. Removed the invalid database constraint.
4. Allowed queued mobile submissions to synchronize at a controlled rate.
5. Reconciled client and server identifiers to identify missing or duplicated records.
6. Refreshed public dashboards after synchronization completed.
7. Contacted affected web participants whose failed submissions could not be reconstructed completely.

API scaling alone did not resolve the incident because retry traffic continued to generate rejected database writes. Service recovered only after retries were disabled and the constraint was removed.

## What Went Well

- Mobile applications retained completed checklists locally, preventing permanent loss for mobile participants.
- Remote configuration allowed retry behavior to be changed without publishing new mobile releases.
- Database logs contained enough constraint metadata to identify the failing migration quickly.
- Regional coordinators helped communicate the outage and advised participants to retain field notes.
- The reconciliation process successfully matched delayed mobile submissions without creating duplicate checklists.

## What Went Poorly

- The data model encoded an invalid assumption about how shared survey sites are used.
- Automated tests did not represent multiple independent observers at the same site.
- Monitoring treated a large increase in `409` responses as non-actionable.
- The deployment occurred too close to a known traffic peak.
- The web experience did not reliably preserve participant input after submission failures.
- Early mitigation focused on capacity rather than the source of rejected writes.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Replace site-based duplicate detection with checklist-scoped idempotency keys. | Data Platform | P0 | May 20, 2026 |
| Add integration tests covering multiple participants, teams, and checklists at the same site and time. | Survey API | P0 | May 18, 2026 |
| Update clients to retry only network errors, timeouts, `429`, and eligible `5xx` responses. | Mobile and Web | P0 | May 27, 2026 |
| Preserve all web-form fields locally until the server confirms submission. | Web Platform | P0 | June 3, 2026 |
| Alert on abnormal rates of all non-successful submission responses, including `409`. | Reliability Engineering | P1 | May 16, 2026 |
| Add retry-storm scenarios to peak-event load tests. | Performance Engineering | P1 | May 31, 2026 |
| Require a survey-domain reviewer for schema changes affecting observation identity or validation. | Engineering Management | P1 | May 15, 2026 |
| Establish a deployment freeze beginning 12 hours before major survey windows. | Release Engineering | P1 | May 14, 2026 |
| Add a feature flag or documented fast rollback path for new database constraints. | Data Platform | P1 | June 7, 2026 |
| Add synchronization-backlog and client-retry metrics to the incident dashboard. | Reliability Engineering | P2 | June 14, 2026 |
| Run a participant communication exercise before the next seasonal count. | Community Operations | P2 | June 21, 2026 |

## Lessons Learned

Database constraints are valuable safeguards only when they reflect the real-world identity rules of the system. In this case, a technically valid migration enforced an incorrect domain assumption and converted legitimate survey activity into rejected writes.

Future changes to observation identity, deduplication, and validation will require representative multi-participant test data and review by both engineering and survey-operations staff. We will also treat retry behavior as part of system capacity planning, particularly for scheduled events where many clients reconnect or submit at the same time.