# Postmortem: Matchmaking and Cloud Save Outage

**Incident date:** July 11, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-1  
**Services affected:** Matchmaking, player authentication, cloud saves  
**Status:** Resolved

## Summary

On July 11, our production backend experienced a database connection exhaustion event during a weekend player-traffic spike. Players were unable to join online matches, and some could not sign in or synchronize cloud saves.

The incident began after a recently deployed matchmaking change increased the number of database queries performed while assembling lobbies. At peak traffic, slow queries held connections longer than expected. Automatic retries from game clients and backend workers amplified the load until the connection pool was exhausted.

We restored service by disabling the new matchmaking logic, reducing retry traffic, and restarting affected backend instances. No confirmed cloud save data was permanently lost, although some players temporarily saw outdated save data until synchronization recovered.

## Customer Impact

During the incident:

- Approximately 68% of matchmaking attempts failed or timed out.
- Approximately 21% of active players experienced sign-in failures.
- Around 4,300 cloud save synchronization requests were delayed.
- 312 players reported seeing older save data after signing in on a second device.
- No purchases or account entitlements were lost.
- No permanent save-data loss has been identified.

Players already in offline or single-player sessions were largely unaffected. Players in active online matches generally remained connected because match servers did not depend on the degraded database after session creation.

## Timeline

All times are in Eastern Time.

- **18:02** — Matchmaking update version `2026.07.11.1` deployed to production.
- **18:17** — Database query latency begins increasing as evening traffic grows.
- **18:31** — First automated alert fires for elevated matchmaking latency.
- **18:34** — On-call engineer acknowledges the alert and begins investigating.
- **18:39** — Matchmaking error rate exceeds 25%. Support receives the first player reports.
- **18:44** — Database connection pool reaches 95% utilization.
- **18:48** — Authentication and cloud save requests begin failing because they share the same database cluster and connection proxy.
- **18:53** — Incident declared SEV-1. Engineering lead and backend engineer join the response.
- **19:01** — Team identifies a sharp increase in lobby-candidate queries but initially suspects a missing database index.
- **19:14** — A new index is prepared but not applied after testing shows it would not address the dominant query pattern.
- **19:22** — Client retry traffic is identified as a major load multiplier.
- **19:31** — Remote configuration increases matchmaking retry intervals from 2 seconds to 15 seconds.
- **19:38** — Incoming request volume drops by approximately 35%, but connection utilization remains critical.
- **19:46** — Engineers correlate the regression with the matchmaking deployment.
- **19:52** — Rollback begins.
- **20:07** — Rollback completes, but several backend instances retain exhausted connection pools.
- **20:14** — A controlled restart of matchmaking and API instances begins.
- **20:29** — Authentication success rate returns to normal.
- **20:36** — Matchmaking success rate recovers above 95%.
- **20:42** — Cloud save backlog begins draining.
- **20:49** — All primary service indicators return to normal.
- **21:07** — Cloud save backlog is fully processed.
- **21:18** — Incident marked resolved after an extended monitoring period.

## Root Cause

The outage was caused by a change to matchmaking lobby assembly introduced in version `2026.07.11.1`.

The new logic evaluated additional compatibility preferences for each potential lobby. Due to an implementation error, the service fetched player preference records separately for every lobby candidate instead of loading them in one batched query. This created an N+1 query pattern: a single matchmaking request that previously required 3 database queries could require more than 80 under common conditions.

The additional queries were individually inexpensive during testing, but at production traffic levels they increased database CPU usage and query latency. Connections remained occupied longer, causing the shared connection pool to fill. Once requests began timing out, automatic retries from clients and background workers generated more requests, further increasing load.

Authentication, matchmaking, and cloud saves used separate application pools but shared the same database proxy and cluster connection limit. Matchmaking therefore consumed capacity needed by otherwise healthy services.

## Contributing Factors

- Load testing used smaller lobby candidate sets than production.
- Performance tests measured request latency but did not enforce a query-count budget.
- Staging traffic did not reproduce production retry behavior.
- Matchmaking clients retried timed-out requests too quickly and without sufficient jitter.
- Multiple critical services shared a database-level connection limit.
- The deployment completed shortly before the studio’s predictable weekend traffic peak.
- The rollback restored the previous code but did not automatically recycle connections held by unhealthy instances.
- The initial alert identified high latency but did not clearly show connection-pool saturation or query amplification.

## Detection and Response

Automated monitoring detected elevated matchmaking latency within 14 minutes of the first measurable degradation. However, the incident was not immediately recognized as a shared database-capacity problem because authentication and cloud save alerts fired several minutes later.

The response was slowed by incomplete per-endpoint query metrics and by an early focus on database indexing. Once engineers compared query volume by application version, the problematic deployment was identified and rolled back.

Remote configuration for client retry behavior reduced pressure without requiring a game update and was important in stabilizing the system during recovery.

## Resolution

Service was restored through the following actions:

1. Increased matchmaking retry intervals and added jitter through remote configuration.
2. Rolled back the matchmaking service to the previous release.
3. Restarted backend instances with exhausted or unhealthy connection pools.
4. Temporarily reserved additional database connections for authentication and cloud saves.
5. Monitored and drained the cloud save synchronization backlog.

## What Went Well

- Remote configuration allowed us to reduce retry traffic without shipping a client patch.
- The previous backend release was available and could be rolled back quickly.
- Cloud save operations were idempotent, preventing duplicate or conflicting writes during backlog recovery.
- Support consolidated player reports into a single incident thread, helping engineering confirm the scope.
- Existing save-history retention allowed us to verify that no permanent save loss occurred.

## What Went Poorly

- The change passed review and testing despite dramatically increasing database query volume.
- Shared database capacity allowed one service to affect unrelated player features.
- Dashboards did not make the N+1 query pattern immediately visible.
- Retry behavior amplified the incident.
- Our public status page was updated 38 minutes after customer impact began.
- Restart requirements after rollback were undocumented, extending recovery time.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Replace per-candidate preference lookups with a batched query | Backend | P0 | July 18, 2026 |
| Add automated query-count limits to matchmaking integration tests | Backend | P0 | July 22, 2026 |
| Add connection-pool saturation and wait-time alerts | Infrastructure | P0 | July 19, 2026 |
| Implement exponential backoff and jitter in all client retry paths | Client | P0 | July 29, 2026 |
| Reserve database capacity for authentication and cloud saves | Infrastructure | P1 | August 5, 2026 |
| Expand load tests to use production-sized candidate sets and retry behavior | QA/Backend | P1 | August 7, 2026 |
| Add automatic rollback triggers for sustained matchmaking error rates | Infrastructure | P1 | August 12, 2026 |
| Document instance restart requirements in the rollback runbook | Backend | P1 | July 18, 2026 |
| Prevent high-risk backend deployments within two hours of peak traffic | Production | P1 | July 17, 2026 |
| Add an incident-status-page update step with a 15-minute target | Community | P1 | July 17, 2026 |
| Evaluate separate database proxies or clusters for critical services | Infrastructure | P2 | August 21, 2026 |
| Run a recovery exercise for matchmaking and cloud save degradation | Engineering | P2 | August 28, 2026 |

## Lessons Learned

For a small studio, sharing infrastructure reduces cost and operational overhead, but it also increases the blast radius of failures. We had treated matchmaking query efficiency as a performance concern when it was also a reliability boundary for authentication and cloud saves.

Future backend changes will be evaluated against explicit query, connection, and retry budgets. We will also prioritize isolation for services required to preserve player access and progress.