# Postmortem: Bird Survey Submission Outage **Incident date:** July 18, 2026 **Duration:** 2 hours 47 minutes **Severity:** SEV-1 **Status:** Resolved **Authors:** Survey Platform Engineering **Last updated:** July 25, 2026 ## Summary On July 18, the bird survey platform experienced a submission outage during the annual summer count. Participants could sign in and view survey routes, but most attempts to submit observations failed or timed out. The incident began after a database migration introduced a synchronous geospatial validation query into the submission path. Under peak load, this query exhausted the primary database connection pool. Automatic retries increased demand, preventing the service from recovering until validation was disabled and application instances were restarted. No confirmed observations were lost after being accepted by the API. However, some participants lost unsaved form data or were unable to submit before their assigned survey window ended. ## Customer Impact Between 08:42 and 11:29 EDT: - 78% of submission attempts failed or timed out. - Approximately 6,400 participants encountered at least one error. - 18,732 submission attempts failed. - An estimated 2,100 participants had to re-enter observations. - 327 offline-mobile submissions were delayed until service recovery. - 146 survey routes received no submission during their scheduled window. Participants saw either a generic “Unable to submit observations” message or a loading indicator that ended after the 30-second client timeout. Drafts saved locally by the mobile application were preserved. The web application did not persist drafts between page reloads, so some web participants lost entered data. ## Timeline All times are in Eastern Daylight Time. - **08:15** — Database migration enabling server-side geospatial boundary validation completed. - **08:30** — Scheduled summer-count participation began increasing. - **08:42** — Submission latency exceeded the five-second service-level objective. - **08:46** — The first automated alert fired for elevated API error rates. - **08:50** — The on-call engineer acknowledged the alert and began investigating. - **08:57** — Support reported multiple participant complaints about failed submissions. - **09:05** — Engineers identified database connection-pool saturation on submission-service instances. - **09:18** — Additional submission-service instances were deployed. Error rates briefly declined, then returned to previous levels. - **09:31** — The incident was declared SEV-1. Database and mobile engineers joined the response. - **09:44** — Query analysis identified the new geospatial validation query as the primary source of long-running transactions. - **10:02** — API retry behavior was found to be amplifying database load. - **10:17** — A configuration change reduced retries from three attempts to zero. Error rates decreased, but submissions remained unreliable. - **10:38** — The team decided to disable synchronous boundary validation and perform it asynchronously. - **10:51** — The validation feature flag was disabled. - **11:03** — Database connections began recovering. Submission-service instances were restarted in stages. - **11:17** — Submission success rate returned above 99%. - **11:29** — Latency and error rates remained stable for ten minutes; the incident was marked resolved. - **13:40** — Delayed offline-mobile submissions finished synchronizing. - **16:15** — Organizers extended affected survey windows and contacted route coordinators. ## Root Cause The immediate cause was a geospatial validation query added to the synchronous observation-submission transaction. The query checked whether each observation coordinate fell within the participant’s assigned survey boundary. In pre-production testing, survey boundaries contained relatively few vertices and database load was low. Production included several highly detailed boundaries with tens of thousands of vertices. Queries involving these boundaries took up to 24 seconds and held database connections for the duration of the transaction. At peak participation, long-running transactions exhausted the submission service’s database connection pool. Requests waiting for connections timed out. The API client then retried failed submissions up to three times, multiplying demand and creating a positive feedback loop. ## Contributing Factors - Load tests used simplified geographic boundaries that did not represent production complexity. - The migration was deployed shortly before a known peak participation period. - The feature flag controlled the user interface but initially did not bypass server-side validation. - Retry logic did not distinguish transient network failures from server overload. - Database alerts monitored CPU and storage but did not directly alert on connection-pool wait time. - The web survey form lacked durable local draft storage. - The runbook recommended scaling application instances for elevated latency, which increased database contention. ## Resolution and Recovery The response team disabled synchronous geospatial validation through a server-side feature flag, reduced automatic retries, and restarted submission-service instances in stages to clear blocked connection pools. Boundary validation was moved to an asynchronous workflow. Submissions are now accepted after basic coordinate validation and marked for later boundary review. Observations outside an assigned route are flagged for coordinator review rather than rejected during submission. After recovery, the team reconciled API logs, mobile synchronization records, and database entries. This confirmed that no submission acknowledged as successful was lost. ## What Went Well - Existing feature-flag infrastructure allowed validation to be disabled without a new application release. - Mobile offline storage preserved queued observations. - Database query tracing made the slow validation query identifiable. - Survey coordinators quickly extended affected observation windows. - Cross-functional response included engineering, support, and survey operations within 49 minutes. ## What Went Poorly - The initial mitigation focused on application capacity rather than the database bottleneck. - Production-like geographic data was absent from performance tests. - Participants received generic errors with no guidance about whether to retry. - Support did not have a prepared incident message for an active survey event. - Web participants could lose drafts when refreshing after a timeout. - Ownership of the geospatial validation feature was unclear during escalation. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---| | Keep geospatial boundary validation outside the synchronous submission path | Survey API Team | P0 | Completed | | Add production-scale boundary fixtures to load and performance tests | Geospatial Team | P0 | August 7, 2026 | | Alert on database connection-pool saturation and wait time | Platform Team | P0 | August 5, 2026 | | Implement exponential backoff, jitter, and retry budgets for submissions | API Team | P0 | August 12, 2026 | | Add idempotency keys to prevent duplicate submissions during retries | API Team | P0 | August 14, 2026 | | Persist web-form drafts in local browser storage | Web Team | P1 | August 21, 2026 | | Show actionable submission errors and preserve entered observations | Web Team | P1 | August 21, 2026 | | Block high-risk migrations within 24 hours of scheduled survey peaks | Engineering Operations | P1 | August 10, 2026 | | Update the incident runbook with database saturation diagnostics | Reliability Team | P1 | August 6, 2026 | | Create event-specific status-page and support-message templates | Support Operations | P1 | August 11, 2026 | | Run a peak-load exercise using anonymized production geometry | Reliability Team | P1 | August 28, 2026 | | Define ownership and rollback procedures for submission-path features | Engineering Management | P2 | August 18, 2026 | ## Lessons Learned Submission availability is more important than synchronous enforcement of correct survey boundaries. Validation that can be reviewed or corrected later should not prevent participants from recording time-sensitive observations. Future readiness reviews for major survey events will include production-shaped geographic data, database connection analysis, retry behavior, draft preservation, and a tested rollback procedure.