# Postmortem: Matchmaking and Login Outage — July 18, 2026

**Incident date:** July 18, 2026  
**Duration:** 2 hours 43 minutes  
**Severity:** SEV-1  
**Services affected:** Authentication, matchmaking, player inventory  
**Status:** Resolved

## Summary

On July 18, our online services experienced an outage that prevented most players from signing in or joining matches. Players already in a match could usually finish it, but post-match rewards were delayed and some inventory screens showed stale data.

The incident began shortly after we deployed a backend update intended to reduce database load during peak hours. A schema migration created an index using a blocking operation on the production player database. As traffic increased, authentication and inventory requests accumulated behind the database lock, exhausting the API connection pool. Automatic retries amplified the load and caused matchmaking requests to fail across all regions.

We restored service by terminating the migration, restarting affected API instances in stages, and temporarily disabling inventory refreshes during login. No permanent player data was lost. Delayed rewards were processed after recovery.

## Customer Impact

Between 19:12 and 21:55 EDT:

- Approximately 78% of login attempts failed or timed out.
- Approximately 64% of matchmaking requests failed.
- Players already in matches could generally continue playing.
- Post-match rewards were delayed for 11,842 accounts.
- Inventory and progression screens displayed stale data for some players.
- 317 players made duplicate cosmetic purchases after receiving timeout messages; these transactions were automatically detected and refunded.
- Support received 436 tickets related to login failures, missing rewards, and duplicate purchases.

The outage affected all platforms and regions. The highest impact occurred in North America and Europe during the evening traffic peak.

## Timeline

All times are in Eastern Daylight Time.

| Time | Event |
|---|---|
| 18:55 | Backend version `2026.07.18.3` deployed to production. |
| 19:04 | Database migration begins on the primary player database. |
| 19:12 | Authentication latency exceeds the alert threshold. |
| 19:15 | On-call engineer acknowledges the alert and begins investigating elevated API error rates. |
| 19:21 | Matchmaking failure rate exceeds 30%. Incident declared SEV-1. |
| 19:27 | Team identifies database connection pool exhaustion across authentication and inventory services. |
| 19:34 | Additional API instances are started, but recovery is minimal because new instances also block on the database. |
| 19:42 | Automated client and service retries increase database connection attempts by approximately 4.6×. |
| 19:51 | Engineers correlate the start of the incident with the active schema migration. |
| 20:02 | Migration is confirmed to be holding a lock on the `player_inventory` table. |
| 20:08 | Migration is terminated. Database latency begins to decline, but API connection pools remain saturated. |
| 20:17 | Authentication and inventory API instances are restarted in stages. |
| 20:31 | Login success rate improves to 71%. Matchmaking remains degraded. |
| 20:39 | Inventory refresh during login is temporarily disabled behind a feature flag. |
| 20:47 | Login success rate returns above 95%. |
| 21:03 | Matchmaking queues begin processing normally in all regions. |
| 21:18 | Reward reconciliation job starts processing delayed match results. |
| 21:42 | Error rates and latency return to normal levels. |
| 21:55 | Incident marked resolved after 30 minutes of stable metrics. |
| 23:26 | Delayed rewards fully reconciled. |
| July 19, 10:30 | Duplicate cosmetic purchases identified and refund process initiated. |

## Root Cause

The direct cause was a production database migration that created an index on the `player_inventory` table using a blocking operation. The table contains approximately 96 million rows, so the migration ran significantly longer than it had in staging.

While the migration held the lock, authentication requests waited on inventory lookups performed during login. These waiting requests consumed all available database connections in the authentication and inventory API pools. Matchmaking depended on the same shared database pool for player entitlement checks, so it also became unavailable.

Client and service retry behavior worsened the incident. Several request paths retried immediately or with insufficient backoff, causing a sharp increase in concurrent database connection attempts.

## Contributing Factors

- The migration was tested against a staging database containing only 4% of production data volume.
- The deployment checklist did not require review of database lock behavior.
- The migration framework defaulted to standard index creation instead of an online or concurrent operation.
- Authentication synchronously loaded inventory metadata even though it was not required to establish a session.
- Authentication, inventory, and matchmaking shared the same database connection limit.
- Retry policies were inconsistent and lacked randomized exponential backoff.
- The migration dashboard showed execution time but did not expose lock waits.
- The initial response focused on adding API capacity, which increased pressure on the constrained database.

## Resolution and Recovery

We terminated the blocking migration and restarted API instances gradually to clear requests holding exhausted connections. We then disabled inventory refreshes during login, allowing authentication to recover without waiting on the inventory table.

After core services stabilized, we ran reconciliation jobs to grant delayed match rewards and identify duplicate purchases. All delayed rewards were applied, and duplicate transactions were refunded to the original payment methods.

The index was later created using a non-blocking operation during a low-traffic maintenance window.

## What Went Well

- Latency and error-rate alerts fired within three minutes of customer impact.
- The on-call engineer declared a SEV-1 early and brought in database support quickly.
- Feature flags allowed us to remove inventory refreshes from the login path without a new deployment.
- Match results were durably queued, preventing loss of progression or rewards.
- Existing transaction identifiers allowed duplicate purchases to be detected and refunded automatically.

## What Went Poorly

- Our staging environment did not represent production database size or query behavior.
- We lacked an alert specifically for long-held database locks.
- The migration could be executed without explicit approval from a second engineer.
- Shared database pools allowed one workload to degrade unrelated services.
- Retry behavior amplified load during an already constrained period.
- The public status page was not updated until 38 minutes after the incident began.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Require non-blocking index creation for production migrations. | Backend Lead | P0 | July 22 |
| Add CI checks that reject known blocking migration patterns. | Platform Engineer | P0 | July 29 |
| Require peer approval and an execution plan for migrations on tables above 10 million rows. | Engineering Manager | P0 | July 22 |
| Add alerts for database lock duration, lock queues, and connection-pool saturation. | Platform Engineer | P0 | July 25 |
| Remove inventory loading from the synchronous login path permanently. | Online Services Team | P1 | August 5 |
| Separate connection pools for authentication, matchmaking, and inventory workloads. | Database Engineer | P1 | August 12 |
| Standardize retries with capped exponential backoff and jitter. | Backend Team | P1 | August 9 |
| Refresh staging with production-scale synthetic data and representative table statistics. | QA and Platform | P1 | August 16 |
| Add a migration cancellation procedure to the incident runbook. | Database Engineer | P1 | July 24 |
| Automate status-page updates when a SEV-1 is declared. | Community Operations | P2 | August 20 |
| Run a failure exercise covering database lock contention and connection exhaustion. | Engineering Manager | P2 | August 30 |

## Prevention

Future production migrations will be evaluated for lock behavior, expected duration, rollback strategy, and performance at production scale. High-risk migrations will run separately from application deployments during monitored maintenance windows.

We are also reducing coupling between authentication and player inventory so that degraded secondary systems do not prevent players from establishing sessions or joining matches. Finally, consistent backoff and workload-specific connection limits will reduce the chance that retries or one overloaded service can exhaust shared database capacity.