# Postmortem: Matchmaking and Inventory Outage **Incident date:** July 12, 2026 **Duration:** 2 hours 17 minutes **Severity:** SEV-1 **Services affected:** Matchmaking, player inventory, progression **Status:** Resolved ## Summary On July 12, our backend experienced a cascading failure that prevented many players from joining online matches and caused delayed inventory and progression updates. The incident began during the rollout of a backend release intended to reduce inventory lookup latency. A database migration created an index on the production inventory table without using the database's non-blocking option. The migration held locks that caused inventory requests to queue. Those requests exhausted the shared application connection pool, which also served matchmaking and progression traffic. Automated retries increased database load and extended the outage. We restored service by stopping the deployment, terminating the migration, restarting affected application instances, and temporarily reducing retry rates. No confirmed inventory or progression data was permanently lost. Some players saw stale balances or delayed rewards until queued events were processed. ## Customer Impact Between 19:08 and 21:25 UTC: - Approximately 61% of matchmaking attempts failed or timed out. - Approximately 34% of active players received inventory errors. - Match rewards and progression updates were delayed for 18,700 accounts. - The longest reward delay was 3 hours and 12 minutes. - 2,340 purchases appeared as pending or missing until reconciliation completed. - No customers were charged more than once. - Single-player gameplay remained available, but cloud synchronization was delayed. - The status page was not updated until 38 minutes after the incident began. ## Timeline All times are in UTC. - **18:55** — Backend version `2026.07.12.3` deployment begins. - **19:03** — Database migration starts creating an index on the inventory table. - **19:08** — Inventory API latency exceeds the alert threshold. - **19:10** — Matchmaking timeout rate begins increasing as the shared connection pool saturates. - **19:12** — Automated alert opens a SEV-2 incident. - **19:16** — On-call engineer acknowledges the alert and begins investigating application instances. - **19:22** — Deployment is paused at 60% completion. - **19:27** — Incident escalated to SEV-1 after matchmaking failures exceed 50%. - **19:31** — Team initially suspects the new inventory cache logic and begins rolling back application code. - **19:39** — Rollback completes, but error rates remain elevated. - **19:44** — Database lock monitoring identifies the migration as the oldest blocking transaction. - **19:49** — Migration is terminated. Database lock contention begins decreasing. - **19:53** — Retry traffic causes database CPU to remain above 95%. - **20:01** — Inventory retries are temporarily limited at the API gateway. - **20:09** — Application connection pools begin recovering. - **20:17** — Matchmaking success rate improves to 70%. - **20:26** — A subset of application instances remains unhealthy due to exhausted pools. - **20:34** — A controlled restart of affected instances begins. - **20:46** — Matchmaking success rate returns above 95%. - **20:58** — Inventory reads return to normal latency. - **21:07** — Queued progression and reward events begin processing normally. - **21:25** — Customer-facing services are considered stable; incident moves to monitoring. - **22:41** — Purchase reconciliation completes. - **00:20** — All delayed reward and progression events are processed. ## Root Cause The direct cause was a production database migration that created a new index on the `player_inventory_items` table using a blocking operation. The migration had been tested against staging, where the table contained approximately 4% of the rows present in production and had little concurrent write traffic. It completed in under a minute and did not expose meaningful lock contention. In production, the index operation ran for several minutes while holding locks that blocked inventory writes and some reads. Inventory requests accumulated and consumed nearly all connections in a pool shared by the inventory, matchmaking, and progression components. Once the pool was saturated, otherwise healthy matchmaking requests could not acquire database connections. Client and server retries then multiplied traffic against the already constrained database, slowing recovery even after the blocking migration was terminated. ## Contributing Factors - The migration framework did not default to non-blocking index creation. - Production-scale migration behavior was not validated before deployment. - Matchmaking and inventory workloads shared the same database connection pool. - Retry policies lacked sufficient exponential backoff and jitter. - The deployment checklist did not require reviewing expected lock behavior. - Database lock alerts were informational and did not page the on-call engineer. - Initial dashboards emphasized application errors but did not prominently display blocking database transactions. - The rollback process reverted application code but did not automatically stop an active migration. - The incident response guide did not clearly assign responsibility for customer communications. ## Resolution and Recovery We terminated the blocking migration and halted the deployment. Because queued retries continued to overload the database, we temporarily reduced retry rates at the gateway and restarted application instances with exhausted connection pools. After service latency normalized, we reconciled purchases against payment-provider records and replayed queued reward and progression events. We verified account balances using event logs and database snapshots. The index change remains disabled pending a revised migration plan. ## What Went Well - Automated monitoring detected elevated inventory latency within two minutes. - Payment idempotency protections prevented duplicate charges. - Reward and progression events were retained in the queue and could be replayed. - Engineers from gameplay services and backend infrastructure joined the response quickly. - Purchase reconciliation tooling worked without requiring manual account edits. ## What Went Poorly - We focused on application rollback for approximately 20 minutes before identifying the database lock. - The status page update was delayed and initially understated matchmaking impact. - Staging data volume and traffic were not representative of production. - The shared connection pool allowed one degraded workload to affect unrelated services. - Retry behavior amplified load during both the incident and recovery. ## Action Items | Action | Owner | Priority | Due Date | |---|---|---:|---| | Change index migrations to use non-blocking creation by default | Backend | P0 | July 23, 2026 | | Add a deployment guard that rejects known blocking production migrations | Platform | P0 | July 30, 2026 | | Separate matchmaking and inventory database connection pools | Backend | P0 | August 6, 2026 | | Add exponential backoff, jitter, and retry budgets to inventory clients | Online Services | P0 | July 27, 2026 | | Page on sustained production database lock waits | Infrastructure | P1 | July 24, 2026 | | Add blocking-transaction panels to the primary incident dashboard | Infrastructure | P1 | July 21, 2026 | | Create a production-scale dataset for migration testing | Data | P1 | August 13, 2026 | | Require lock-impact and rollback analysis in migration reviews | Engineering | P1 | July 22, 2026 | | Update deployment tooling so rollbacks stop active migrations when safe | Platform | P1 | August 10, 2026 | | Run a load test covering connection-pool exhaustion and retry storms | QA | P1 | August 17, 2026 | | Add status-page ownership and update intervals to the incident guide | Operations | P2 | July 20, 2026 | | Conduct a database failure-mode exercise with the on-call rotation | Engineering | P2 | August 24, 2026 | ## Lessons Learned For a small team, shared infrastructure reduces operational overhead, but it also increases the blast radius of failures. We treated this migration as a routine schema change because it was fast in staging. Production data size and concurrency made its behavior fundamentally different. Future migrations affecting large or frequently written tables will be treated as production operations with explicit lock analysis, load testing, monitoring, and abort criteria. We are also isolating critical matchmaking capacity so inventory degradation cannot prevent players from joining games.