# Postmortem: Matchmaking Outage During Season Launch **Incident date:** July 24, 2026 **Duration:** 1 hour 47 minutes **Severity:** SEV-1 **Services affected:** Matchmaking, party creation, player authentication **Status:** Resolved ## Summary During the launch of Season 4, players experienced failed logins, long matchmaking queues, and repeated party disconnections. The incident began when increased traffic caused our matchmaking workers to exhaust the game-session database connection pool. An automatic scaling rule added more workers in response to the growing queue. Because every worker opened its own database connection pool, scaling increased pressure on the database rather than relieving it. The database reached its connection limit, and dependent authentication and party services subsequently began timing out. We restored service by limiting the number of matchmaking workers, restarting saturated services in stages, and temporarily increasing database connection capacity. ## Customer Impact From 18:06 to 19:53 UTC: - Approximately 38% of login attempts failed or timed out. - Approximately 61% of matchmaking requests took longer than 90 seconds. - Approximately 24% of active parties were disconnected. - 17,400 unique players encountered at least one error. - 312 matches ended early after session-heartbeat updates failed. - No player inventory, progression, or purchase data was lost. - Store purchases were temporarily unavailable because authentication could not reliably validate sessions. Players already in matches were initially unaffected, but some were disconnected once heartbeat writes began timing out. The outage was most severe in North America and Western Europe, where launch traffic peaked simultaneously. ## Timeline All times are in UTC on July 24, 2026. - **17:45** — Season 4 deployment completes. Initial health checks pass. - **18:00** — Scheduled season launch begins. Concurrent player count rises faster than forecast. - **18:06** — Matchmaking latency exceeds the alert threshold. - **18:08** — Autoscaling increases matchmaking workers from 24 to 48. - **18:11** — Database connections reach 92% of the configured limit. - **18:13** — Authentication and party-service requests begin timing out. - **18:15** — On-call engineer acknowledges the matchmaking-latency alert. - **18:19** — Support reports a sharp increase in login and party-disconnection complaints. - **18:23** — Incident declared SEV-1. Engineering lead assumes incident command. - **18:27** — Team initially suspects the newly deployed matchmaking ranking changes and begins reviewing application errors. - **18:34** — Database dashboard shows connection exhaustion, but the dashboard incorrectly attributes most connections to a shared service account. - **18:40** — Matchmaking worker count reaches 72. Database connection limit is reached. - **18:43** — Autoscaling is disabled. Worker count is manually reduced to 30. - **18:49** — Database connections begin declining, but authentication remains degraded due to retry backlogs. - **18:56** — Engineers disable nonessential leaderboard and telemetry writes. - **19:04** — Database connection capacity is temporarily increased from 800 to 1,200. - **19:12** — Authentication success rate returns above 95%. - **19:18** — Matchmaking queue begins draining. - **19:27** — Party creation and reconnection rates return to normal. - **19:34** — Matchmaking worker count is cautiously increased to 36. - **19:41** — Error rates and latency remain within normal ranges. - **19:53** — Incident resolved. Season launch remains active under increased monitoring. - **21:10** — Temporary database capacity increase is retained pending a permanent fix. ## Root Cause Each matchmaking worker maintained a database connection pool with a maximum of 20 connections. The autoscaling configuration treated workers as stateless and scaled them based only on queue depth and CPU utilization. At launch, a higher-than-expected number of players entered matchmaking within a short period. Queue depth increased, causing the platform to add workers. Every new worker created another connection pool, so scaling from 24 to 72 workers increased the potential matchmaking connection count from 480 to 1,440. The database cluster was configured to accept a maximum of 800 connections across all services. Once the limit was reached, matchmaking queries timed out. Authentication, party management, and session-heartbeat requests used the same database cluster and were unable to acquire connections. Their built-in retries added more traffic and prolonged recovery. The Season 4 application release did not introduce the faulty behavior. However, the season launch generated enough traffic to expose an existing capacity and architecture issue. ## Contributing Factors - Load tests modeled gradual player growth rather than a synchronized launch spike. - Staging used fewer workers and a proportionally larger database connection allowance, so connection exhaustion did not appear during testing. - Autoscaling had no database-connection or dependency-health guardrail. - Multiple services shared one database cluster and connection limit. - All services used the same database account, making connection ownership difficult to identify. - Retry policies lacked jitter and used aggressive timeouts, creating a retry surge. - The alert for database connection utilization fired at 90%, leaving little time to intervene. - The runbook described increasing worker count as the standard response to matchmaking queue growth. - Launch capacity estimates were based on the previous season and did not account for the recent console promotion. ## Resolution and Recovery We disabled matchmaking autoscaling and reduced the worker count until the database could accept new connections. We then disabled nonessential leaderboard and telemetry writes to free additional capacity. After confirming that connection exhaustion was the primary failure mode, we temporarily increased the database connection limit. Authentication and party services were restarted in stages to prevent all instances from reconnecting simultaneously. Matchmaking workers were added back gradually while engineers monitored database utilization and request latency. ## What Went Well - Existing inventory and progression safeguards prevented data loss. - Support escalated player reports quickly, helping confirm the scope of the incident. - The incident commander established clear ownership once the event was declared SEV-1. - Disabling nonessential writes reduced pressure without requiring a rollback. - Staged service recovery prevented a second connection spike. ## What Went Poorly - The first alert identified matchmaking latency but not the underlying database saturation. - Engineers spent approximately 20 minutes investigating the Season 4 release before focusing on connection usage. - Database telemetry could not distinguish connections by service. - The matchmaking runbook recommended an action that worsened the incident. - Our public status page was not updated until 36 minutes after customer impact began. - We had no tested procedure for reducing workers while preserving queued matchmaking requests. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---| | Reduce matchmaking connection-pool size and enforce a global connection budget | Backend | P0 | August 3, 2026 | | Add autoscaling guardrails based on database saturation and dependency error rates | Platform | P0 | August 7, 2026 | | Assign separate database users to each service for connection attribution | Platform | P0 | August 5, 2026 | | Add exponential backoff and jitter to database retries | Backend | P0 | August 10, 2026 | | Run a launch-spike load test at 150% of forecast concurrency | QA/Platform | P0 | August 14, 2026 | | Alert at 70% and 80% database connection utilization | SRE | P1 | August 1, 2026 | | Update the matchmaking incident runbook | SRE | P1 | August 4, 2026 | | Automate status-page updates when a SEV-1 incident is declared | Community/Platform | P1 | August 21, 2026 | | Isolate session-heartbeat traffic from matchmaking queries | Backend | P1 | September 4, 2026 | | Review database capacity before every scheduled content launch | Engineering Lead | P1 | Ongoing | | Add a launch-day capacity checklist and named incident roles | Production | P2 | August 12, 2026 | | Evaluate a connection proxy to smooth worker scaling | Platform | P2 | September 18, 2026 | ## Lessons Learned Autoscaling a service does not increase capacity when the service is constrained by a shared dependency. Scaling policies must account for database limits, connection behavior, and downstream health—not only CPU usage and queue depth. Future launch tests will model synchronized player arrivals and include every production service competing for shared resources. We will also treat public communication as part of incident response, with explicit ownership assigned at the beginning of each major launch.