# Incident Postmortem: Bird Survey Submission Outage

**Date:** 2026-06-14  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-2  
**Status:** Resolved

## Summary

During the first morning of the Spring Breeding Bird Survey weekend, the survey platform experienced a partial outage that prevented many volunteers from submitting checklists through the mobile app and web portal. Users could still view routes and saved drafts, but submission attempts failed or timed out.

The outage was caused by a database migration that added a required field to survey submissions before all application servers had been updated to populate that field.

## Customer Impact

- Approximately 3,800 volunteers were affected.
- 12,460 checklist submissions failed on first attempt.
- 2,910 users were unable to submit during their active survey window.
- No submitted observations were lost.
- Drafts saved locally in the mobile app were retained and successfully synced after service recovery.
- Regional coordinators experienced delays validating same-day survey coverage.

## Timeline

All times in EDT.

| Time | Event |
| --- | --- |
| 05:30 | Scheduled database migration begins. |
| 05:42 | Migration completes successfully. |
| 05:48 | First increase in submission errors begins. |
| 06:03 | Monitoring alert fires for elevated API 500 responses. |
| 06:07 | On-call engineer begins investigation. |
| 06:18 | Support team reports multiple volunteer complaints about failed checklist submissions. |
| 06:26 | Engineers identify failures concentrated on `POST /survey-submissions`. |
| 06:41 | Root cause traced to application servers not sending newly required `observer_effort_minutes` field. |
| 06:55 | Database constraint is temporarily relaxed. |
| 07:03 | Submission success rate begins recovering. |
| 07:22 | Backlog of queued mobile submissions starts processing normally. |
| 07:47 | All API error rates return to baseline. |
| 08:05 | Incident declared resolved. |

## Root Cause

The outage was caused by an unsafe schema migration.

A new column, `observer_effort_minutes`, was added to the `survey_submissions` table with a `NOT NULL` constraint. The migration was deployed before the corresponding application release had fully rolled out to all API servers.

Older application instances continued accepting submissions without the new field. When those requests reached the database, inserts failed due to the constraint violation. Because submission retries used the same invalid payload, affected users saw repeated failures until the database constraint was relaxed.

## Contributing Factors

- The migration combined schema creation and constraint enforcement in a single deployment.
- The deployment checklist did not require backward compatibility verification for mobile and API clients.
- Canary monitoring focused on read traffic and did not include synthetic survey submissions.
- The incident occurred during a known high-traffic survey window.
- Error messages shown to users were generic and did not explain that drafts were safe.

## Resolution

Engineers removed the `NOT NULL` constraint from `observer_effort_minutes`, allowing submissions from older application instances to succeed. Once all API servers were confirmed to be running the updated version, missing effort values were backfilled using survey start and end timestamps where available.

## Action Items

| Action Item | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Split schema changes into expand, backfill, and enforce phases. | Platform | 2026-06-28 | Open |
| Add migration review checklist for backward compatibility. | Engineering | 2026-06-21 | Open |
| Add synthetic submission monitoring for mobile and web flows. | SRE | 2026-06-30 | Open |
| Block non-emergency migrations during national survey windows. | Release Management | 2026-06-21 | Open |
| Improve user-facing sync errors to clarify that drafts are retained. | Mobile | 2026-07-05 | Open |
| Add automated tests for old-client submission compatibility. | API | 2026-06-28 | Open |

## Lessons Learned

Schema migrations that enforce new required fields must be deployed only after all writers have been updated and verified. For field programs with narrow observation windows, even short outages can create operational problems for volunteers and coordinators. Future releases will treat survey weekends as high-risk periods and require stronger compatibility checks before production changes.