# Postmortem: Matchmaking and Progression Outage **Incident date:** July 12, 2026 **Duration:** 2 hours 17 minutes **Severity:** SEV-1 **Services affected:** Matchmaking, player inventory, progression saves **Status:** Resolved ## Summary On July 12, our multiplayer backend experienced a database connection storm after a routine game-server deployment. Players could launch the game, but many could not join matches. Some completed matches without receiving rewards or progression updates. The outage lasted 2 hours and 17 minutes. Matchmaking availability fell to 31% at the incident’s peak, and progression writes were delayed for approximately 18,400 accounts. No permanent inventory or progression data was lost. ## Customer Impact Between 18:42 and 20:59 UTC: - Approximately 27,600 players encountered at least one matchmaking failure. - 8,900 players were disconnected while joining or completing a match. - 18,400 accounts experienced delayed XP, currency, achievement, or inventory updates. - 1,240 purchases appeared missing until delayed processing completed. - Support received 763 tickets related to matchmaking, missing rewards, or purchases. All queued progression and purchase events were replayed by 23:26 UTC. We verified that no completed purchases or acknowledged progression events were permanently lost. ## Timeline All times are in UTC. - **18:30** — Version 1.8.4 game servers begin rolling deployment. - **18:39** — The first updated servers enter the matchmaking pool. - **18:42** — Database connection count rises sharply; matchmaking latency alerts fire. - **18:45** — Automated scaling adds matchmaking workers, increasing connection pressure. - **18:48** — Matchmaking success rate falls below 70%. - **18:52** — On-call engineer acknowledges the alert and begins investigation. - **19:01** — Database reaches its connection limit. Progression and inventory writes begin timing out. - **19:08** — Incident is declared SEV-1. The deployment is paused at 63%. - **19:17** — The team identifies elevated connection creation from updated game servers. - **19:24** — A rollback begins, but several servers fail to drain active sessions cleanly. - **19:39** — Rollback completes. Connection count remains elevated because unhealthy server processes continue retrying. - **19:47** — Matchmaking worker autoscaling is disabled to stop additional database connections. - **19:55** — Retry traffic is rate-limited, and stuck game-server processes are terminated. - **20:08** — Database connection usage begins returning to normal. - **20:21** — Matchmaking success rate recovers above 90%. - **20:34** — Progression writes resume. Backlogged events begin processing. - **20:51** — Matchmaking and purchase validation metrics return to baseline. - **20:59** — Incident is resolved. Monitoring continues while progression events replay. - **23:26** — Backlog processing completes and account reconciliation passes. ## Root Cause Version 1.8.4 introduced a change to the game-server database client intended to isolate requests by match. The change accidentally created a new connection pool for every active match instead of one pool per server process. Under production concurrency, each updated game server opened hundreds of database connections. The database reached its configured connection limit, causing matchmaking, inventory, and progression requests to time out. The outage was amplified by two automated recovery mechanisms: 1. Matchmaking workers scaled out in response to rising latency, creating additional database connections. 2. Game servers retried failed writes without exponential backoff or jitter, producing a synchronized retry storm. ## Contributing Factors - Load tests simulated player requests but did not reproduce the number of concurrent matches per server. - Staging used a smaller topology and a database proxy with different connection behavior. - Connection-count alerts were configured at the database level but not attributed by application version or service. - The deployment health check measured process availability, not matchmaking success or database connection growth. - The rollback procedure did not immediately terminate unhealthy processes that failed to drain. - The client library allowed connection pools to be instantiated without a process-level limit. ## Resolution and Recovery We rolled back version 1.8.4, disabled matchmaking autoscaling, rate-limited retries, and terminated remaining processes running the affected build. Once database connection usage stabilized, we restored matchmaking capacity gradually. Progression and purchase events are written to a durable queue before acknowledgement. After database recovery, we replayed the backlog and ran reconciliation against platform purchase receipts and match-completion records. ## What Went Well - Durable event queuing prevented permanent progression and purchase loss. - Existing latency alerts detected the issue within three minutes. - The small team quickly reassigned engineers from client and gameplay work to incident response. - Account reconciliation tooling confirmed recovery without requiring manual edits. ## What Went Poorly - The deployment continued for several minutes after database connections began rising. - Autoscaling treated database saturation as insufficient application capacity. - Our first rollback did not remove all affected processes. - The public status page was not updated until 37 minutes after impact began. - Support lacked an approved response for delayed purchases and progression. ## Action Items | Action | Owner | Priority | Due Date | |---|---|---:|---| | Enforce one database connection pool per server process | Backend | P0 | July 17 | | Add a hard per-process connection limit to the database client | Backend | P0 | July 19 | | Add exponential backoff, jitter, and retry budgets for writes | Backend | P0 | July 22 | | Gate deployments on matchmaking success rate and connection growth | Infrastructure | P0 | July 24 | | Add production-scale concurrent-match tests | QA/Backend | P1 | July 29 | | Attribute database connections by service and release version | Infrastructure | P1 | July 26 | | Prevent autoscaling when the database is saturated | Infrastructure | P1 | July 31 | | Update rollback automation to terminate processes that fail to drain | Infrastructure | P1 | July 28 | | Align staging database proxy behavior with production | Infrastructure | P1 | August 5 | | Automate status-page updates for confirmed SEV-1 incidents | Operations | P2 | August 9 | | Create support templates for delayed rewards and purchases | Player Support | P2 | July 20 | | Run a database-saturation incident exercise | Engineering | P2 | August 15 | ## Prevention Future backend releases will use staged traffic percentages with automatic rollback based on customer-facing success metrics. Database connection usage will become a deployment gate, and all services will use bounded connection pools and standardized retry policies. We will review the effectiveness of these changes after the next two production releases and publish follow-up findings internally.