# Postmortem: Matchmaking and Player Progression Outage

**Incident date:** July 11, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Services affected:** Matchmaking, player inventory, progression saves  
**Status:** Resolved

## Summary

On July 11, our production backend experienced a cascading failure that prevented most players from joining online matches. Players already in matches could continue playing, but progression and inventory updates were delayed. A small subset of completed matches initially appeared to lose rewards; all recoverable rewards were restored from the event log within 24 hours.

The incident began during deployment of a backend release intended to reduce matchmaking queue times. A database migration added an index to the production `player_sessions` table without using an online or concurrent operation. The migration held a lock that blocked session writes. Application instances retried those writes without sufficient backoff, exhausting the database connection pool and causing failures in matchmaking and progression services that shared the same database.

We restored service by stopping the deployment, terminating the migration, disabling automatic retries for the affected operation, and gradually restarting application instances.

## Customer Impact

From 19:14 to 22:01 EDT:

- Approximately 78% of matchmaking attempts failed or timed out.
- Roughly 6,400 players received “Unable to connect” or “Matchmaking unavailable” errors.
- About 1,150 completed matches had delayed progression or inventory updates.
- 214 players temporarily saw missing rewards after restarting the game.
- No purchases, account credentials, or permanent player records were lost.
- All but nine reward records were restored automatically from the event log. The remaining nine were corrected manually after reviewing match telemetry.

Our status page was updated 38 minutes after the incident began. Support received 327 tickets and approximately 900 reports through Discord and social media.

## Timeline

All times are in EDT.

| Time | Event |
|---|---|
| 19:05 | Backend version `2026.07.11.3` deployment begins at 10% capacity. |
| 19:11 | Database migration starts creating an index on `player_sessions`. |
| 19:14 | Session-write latency rises sharply as the migration blocks writes. |
| 19:16 | Matchmaking error rate exceeds 20%. An automated alert is sent to the on-call engineer. |
| 19:20 | Deployment automatically pauses because health checks fail. The migration continues running. |
| 19:24 | On-call engineer begins investigation and initially suspects a regional hosting-provider issue. |
| 19:29 | Application retries saturate the shared database connection pool. Progression writes begin timing out. |
| 19:36 | Second engineer joins. The team identifies elevated database lock waits but does not yet connect them to the migration. |
| 19:44 | Incident is declared SEV-1. |
| 19:52 | Public status page is updated. |
| 20:03 | The active migration is identified as the lock holder. |
| 20:08 | Migration process is terminated. Lock contention drops, but retry traffic keeps the database overloaded. |
| 20:17 | Matchmaking workers are scaled down to reduce database pressure. |
| 20:26 | Retry behavior is disabled through a runtime configuration change. |
| 20:34 | Database connections and query latency begin returning to normal. |
| 20:42 | Application instances are restarted in small batches. |
| 21:06 | Matchmaking success rate recovers to 85%. |
| 21:24 | Matchmaking success rate exceeds 98%; progression writes remain delayed. |
| 21:39 | Backlogged progression events begin replaying from the event log. |
| 22:01 | All customer-facing services are stable. Incident is marked mitigated. |
| 23:18 | Automated reward reconciliation completes. Nine records are flagged for manual review. |
| July 12, 15:40 | Manual reward corrections finish. |

## Root Cause

The direct cause was a schema migration that created an index on the production `player_sessions` table using a blocking database operation. The table receives frequent writes for logins, matchmaking reservations, and session heartbeats. While the index was being built, those writes queued behind an exclusive lock.

The release had been tested against a staging database, but staging contained less than 3% of the production row count and had almost no concurrent write traffic. The migration completed in under four seconds in staging, so its locking behavior was not noticed.

The outage became broader because the affected application code retried failed session writes immediately, up to eight times per request. As more requests blocked, retries consumed the shared database connection pool. Matchmaking, inventory, and progression services all depended on that pool, so failure in the session path exhausted resources needed by otherwise healthy operations.

## Contributing Factors

- Database migrations were executed automatically as part of application startup.
- The migration review checklist did not require explicit evaluation of table locks or production data volume.
- Staging data volume and traffic patterns were not representative of production.
- Retry logic used fixed, short delays without jitter or a total retry budget.
- Matchmaking and progression shared one database connection pool.
- The deployment controller paused application rollout but could not cancel an already-running migration.
- The initial alert emphasized HTTP error rate and did not include database lock information.
- The runbook did not describe how to identify and terminate a blocking migration.
- The status page update depended on a developer with separate credentials, delaying communication.

## Detection and Response

Automated monitoring detected the elevated matchmaking error rate within two minutes. However, the first alerts did not clearly indicate database lock contention, and the team spent approximately 15 minutes investigating network and hosting-provider health.

Once the migration was identified, terminating it removed the original lock. Recovery took an additional 26 minutes because accumulated retries continued overwhelming the database. Scaling down workers and disabling retries allowed the connection pool to recover.

The event log for match completion operated as designed and preserved the data needed to reconstruct delayed progression and reward updates.

## Resolution and Recovery

We took the following steps to restore service:

1. Stopped the production deployment.
2. Terminated the blocking index migration.
3. Reduced matchmaking worker capacity to lower write pressure.
4. Disabled retries for session writes through runtime configuration.
5. Restarted application instances gradually to clear blocked connections.
6. Replayed progression events from the durable event log.
7. Reconciled match results against player reward records and manually corrected unresolved cases.

The index change was removed from the release. It will be reapplied using an online operation during a monitored maintenance window.

## What Went Well

- Error-rate monitoring detected the customer-facing failure quickly.
- The deployment system paused further rollout automatically.
- Durable match-completion events prevented permanent loss of most progression data.
- Engineers from backend, operations, and community support coordinated effectively after the incident was declared.
- Runtime configuration allowed retry behavior to be disabled without preparing a new build.

## What Went Poorly

- A blocking schema change reached production without a database-specific safety review.
- Staging did not reveal the migration’s production behavior.
- Aggressive retries amplified a localized lock into a multi-service outage.
- Shared database resources increased the blast radius.
- Diagnosis was delayed by incomplete alerts and an outdated runbook.
- Players received no official communication for the first 38 minutes.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Remove automatic schema migrations from application startup and require a separate deployment step. | Backend Lead | P0 | July 22 |
| Add CI checks that reject blocking index creation on production-sized tables. | Platform Engineer | P0 | July 25 |
| Require lock, runtime, rollback, and data-volume analysis in migration reviews. | Engineering Manager | P0 | July 18 |
| Replace fixed retries with exponential backoff, jitter, and per-request retry budgets. | Backend Team | P0 | July 24 |
| Add circuit breakers around session persistence failures. | Backend Team | P1 | August 2 |
| Separate matchmaking and progression database connection pools. | Platform Engineer | P1 | August 7 |
| Build a production-scale, anonymized staging dataset and continuous write-load test. | QA and Platform | P1 | August 14 |
| Alert directly on long-running locks, connection-pool saturation, and retry volume. | Platform Engineer | P1 | July 21 |
| Update the database incident runbook with lock identification and termination procedures. | On-Call Coordinator | P1 | July 19 |
| Make status-page access available to all incident commanders and support leads. | Studio Operations | P1 | July 17 |
| Add automated reconciliation reporting for delayed progression events. | Data Engineer | P2 | August 12 |
| Run a game-day exercise covering database lock contention and retry storms. | Engineering Manager | P2 | August 21 |

## Lessons Learned

For a small studio, automation reduces operational load, but combining application deployment and schema migration made a risky database operation too easy to execute. Migration safety must be evaluated against production scale and concurrency, not only whether the statement succeeds in staging.

Retries also need to be treated as additional load, not a harmless reliability feature. Without backoff and limits, they can turn temporary contention into sustained resource exhaustion. We will address both issues through safer deployment controls, representative load testing, and isolation between critical backend paths.