# Postmortem: Matchmaking Outage During Season Launch

**Incident date:** July 11, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Services affected:** Matchmaking, party creation, player inventory  
**Status:** Resolved

## Summary

On July 11, shortly after launching Season 4 of *Starfall Brigade*, players began experiencing matchmaking failures, party disconnections, and delayed inventory updates. At the peak of the incident, 78% of matchmaking requests failed.

The outage was caused by an unbounded cache-refresh job introduced with the season update. The job issued concurrent reads against the primary player database for every active item definition. Higher-than-forecast launch traffic amplified the load, exhausting the database connection pool and causing dependent services to time out.

We restored service by disabling the refresh job, restarting affected service instances, and temporarily increasing database capacity. No permanent player-data loss occurred, although some inventory rewards were delayed.

## Customer Impact

Between 18:03 and 20:20 UTC:

- Approximately 41,000 players encountered at least one failed matchmaking attempt.
- 12,600 active parties were disconnected or unable to queue.
- 8,400 inventory updates were delayed by up to 94 minutes.
- 317 purchases initially appeared incomplete; all were reconciled automatically after service recovery.
- Players already inside matches were generally unaffected, but post-match rewards were delayed.
- The studio received 2,100 support requests and a significant increase in negative reviews and social-media reports.

## Timeline

All times are in UTC.

- **17:55** — Season 4 deployment completed across production.
- **18:00** — Seasonal content became available.
- **18:03** — Database connection usage increased from 44% to 91%.
- **18:06** — Matchmaking error rate exceeded 20%.
- **18:08** — Automated alert fired for elevated API latency.
- **18:12** — On-call engineer acknowledged the alert and began investigating.
- **18:17** — Support reported widespread party and matchmaking failures.
- **18:22** — Incident declared SEV-1; engineering, operations, community, and support leads joined the incident channel.
- **18:29** — Team identified connection-pool exhaustion on the primary player database.
- **18:36** — Database capacity was temporarily increased, reducing errors for several minutes.
- **18:43** — Error rates rose again as additional game servers entered the cache-refresh cycle.
- **18:51** — Engineers correlated database load with the new item-definition refresh job.
- **19:02** — Feature flag used to disable the job, but several older service instances did not receive the flag update.
- **19:14** — Rolling restart began to force consistent configuration.
- **19:31** — Database connection usage fell below 70%.
- **19:39** — Matchmaking success rate recovered above 95%.
- **19:48** — Delayed inventory and purchase events began processing from the queue.
- **20:05** — Matchmaking and party services returned to normal operation.
- **20:20** — Inventory backlog cleared; incident moved to monitoring.
- **22:10** — Purchase reconciliation confirmed complete with no permanent data loss.

## Root Cause

The season update added a cache-refresh job for item definitions. The job was intended to run once per deployment and load the current seasonal item set into memory.

Due to a configuration error, the job ran independently on every game-service instance every five minutes. It also created one database query per item definition without a concurrency limit. Season 4 increased the number of definitions from approximately 900 to 3,700.

During launch traffic, 64 service instances each attempted thousands of concurrent reads against the primary player database. These requests consumed the available connections, blocking matchmaking, party, and inventory operations that shared the same database cluster.

The temporary database scale-up did not fully resolve the incident because the refresh job immediately consumed the additional capacity.

## Contributing Factors

- The job was tested with a production-sized item catalog but not with production instance counts.
- The staging environment used only four service instances and did not reproduce the connection pressure.
- Matchmaking and catalog reads shared the same database cluster and connection budget.
- The refresh job lacked concurrency limits, jitter, and leader election.
- The feature flag client did not reliably refresh flags on older service instances.
- Database alerts focused on CPU and storage latency; connection saturation alerted later than it should have.
- The launch traffic forecast underestimated concurrent players by 38%.
- The deployment checklist did not require a database-load review for background jobs.

## Resolution and Recovery

Engineers disabled the cache-refresh job and restarted all game-service instances to ensure the configuration was applied. Database capacity was temporarily increased while queued inventory and purchase events were processed.

Matchmaking recovered as database connections became available. Inventory and commerce events were replayed from the durable queue. A reconciliation job compared platform purchase records with player inventories and granted any missing entitlements.

## What Went Well

- Automated latency monitoring detected the incident before support escalation.
- Durable queues prevented inventory and purchase events from being lost.
- Engineering, support, and community teams coordinated in a single incident channel.
- The team identified the offending job within 48 minutes of the first alert.
- Existing purchase-reconciliation tooling resolved all incomplete entitlements without manual grants.

## What Went Poorly

- The first mitigation addressed database capacity rather than the source of load.
- The feature flag did not propagate consistently, extending the outage.
- We lacked per-job database metrics, making the refresh job difficult to identify.
- Public status updates began 31 minutes after widespread failures started.
- The on-call engineer had no runbook for database connection exhaustion.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---:|
| Replace per-instance refresh with a single leader-elected job | Backend | P0 | July 18 |
| Add bounded concurrency and request jitter to all catalog jobs | Backend | P0 | July 18 |
| Move catalog reads to a dedicated read replica | Infrastructure | P0 | July 25 |
| Add connection-pool budgets per service and workload | Infrastructure | P0 | July 25 |
| Fix feature flag refresh behavior and add propagation monitoring | Platform | P0 | July 22 |
| Add alerts for database connection saturation above 75% | SRE | P1 | July 17 |
| Add per-job query volume and latency dashboards | SRE | P1 | July 24 |
| Create a database connection-exhaustion runbook | SRE | P1 | July 21 |
| Load-test seasonal launches using production instance counts and catalog sizes | QA | P1 | August 1 |
| Require capacity review for new scheduled and background jobs | Engineering Management | P1 | July 29 |
| Separate matchmaking persistence from inventory workloads | Architecture | P2 | September 15 |
| Automate status-page updates from SEV-1 incident creation | Community/Platform | P2 | August 8 |

## Lessons Learned

Background jobs must be treated as production traffic, especially when they scale with both content size and instance count. Increasing capacity can provide temporary relief, but it is not an effective mitigation when a workload has no concurrency controls.

Future seasonal launches will include full-scale load testing, explicit database connection budgets, and a review of every new scheduled job before deployment.