# Postmortem: Bird Survey Submission Outage

Date: 2026-06-22  
Duration: 2 hours 17 minutes  
Severity: SEV-2  
Status: Resolved

## Summary

On June 22, the citizen-science bird survey platform was unable to accept new checklist submissions for most users. Participants could still log in, view survey locations, and draft observations, but attempts to submit completed surveys failed with intermittent `500` errors.

The outage was caused by a database migration that added a required `survey_protocol_version` field without backfilling older mobile clients or making the field temporarily nullable.

## Customer Impact

- 68% of submission attempts failed during the incident.
- Approximately 4,300 bird observation checklists were not submitted successfully.
- 1,100 mobile users saw repeated retry failures.
- Regional survey coordinators could not monitor real-time participation rates.
- No observation data was lost for users who had local draft storage enabled.
- Web users without saved drafts may have needed to re-enter survey data.

## Timeline

All times in UTC.

| Time | Event |
|---|---|
| 13:04 | Database migration deployed to production. |
| 13:07 | Error rate for `POST /api/surveys` increased above baseline. |
| 13:12 | First user reports submission failures through support. |
| 13:18 | On-call engineer paged by API error-rate alert. |
| 13:25 | Initial investigation identified validation failures in survey submission logs. |
| 13:39 | Team confirmed failures were concentrated on older mobile app versions. |
| 13:52 | Migration identified as likely trigger. |
| 14:08 | Hotfix prepared to default missing `survey_protocol_version` values server-side. |
| 14:21 | Hotfix deployed to production. |
| 14:27 | Submission success rate recovered to normal levels. |
| 15:21 | Backfill completed for affected records and queued submissions processed. |

## Root Cause

The root cause was an unsafe schema and application rollout sequence.

A new required field, `survey_protocol_version`, was introduced to support upcoming protocol-specific reporting. The database migration enforced `NOT NULL`, but older mobile clients did not send this field. The API validation layer passed these submissions through, but the database rejected them at write time.

The deployment assumed all active clients would include the new field, which was incorrect.

## Contributing Factors

- The migration was not staged as nullable-first, backfill-second, enforce-last.
- Mobile client compatibility was not tested against the production API contract.
- API validation did not provide a clear client-facing error.
- The alert fired on aggregate API errors, but not specifically on survey submission failure rate.
- The release checklist did not include offline/mobile client version compatibility.

## Resolution

The team deployed a server-side fallback that assigns the current default protocol version when older clients omit the field. Failed queued submissions from mobile clients were retried successfully after the fix.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---:|---|
| Add nullable-first migration policy to engineering release checklist. | Platform | 2026-07-05 | Open |
| Add contract tests for supported mobile client versions. | Mobile/API | 2026-07-12 | Open |
| Create submission-specific alert for failure rate above 5%. | SRE | 2026-07-08 | Open |
| Improve API error messages for validation and persistence failures. | API | 2026-07-15 | Open |
| Add automated migration review for new `NOT NULL` columns. | Data Platform | 2026-07-19 | Open |
| Publish participant-facing incident summary. | Community Team | 2026-06-24 | Complete |

## Lessons Learned

Schema changes that affect public clients must assume mixed client versions for an extended period. Future changes to required submission fields will be rolled out in phases with explicit backward compatibility testing before enforcement.