# Postmortem: Matchmaking and Inventory Outage

**Date of incident:** 2026-06-18  
**Duration:** 1 hour 47 minutes  
**Services affected:** Matchmaking, player inventory, cosmetic unlocks  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On June 18, 2026, players of *Moonforge Raiders* were unable to reliably enter multiplayer matches, equip cosmetics, or claim post-match rewards. The outage began shortly after we deployed backend version `2026.06.18.1`, which included a change to how player inventory data is cached.

A database migration added a new `inventory_revision` field, but one matchmaking service instance continued running an older build that did not include the field in its cache invalidation logic. This caused stale inventory records to be written back to Redis, which then triggered repeated validation failures in the matchmaking flow.

The incident was resolved by draining the stale service instance, clearing affected Redis keys, and redeploying the backend with stricter startup checks.

## Customer Impact

- Approximately 18,400 players were affected.
- 62% of matchmaking attempts failed during the incident window.
- Some players saw cosmetics disappear temporarily from their loadouts.
- Post-match reward claims were delayed for roughly 9,700 players.
- No permanent inventory or currency loss occurred.
- All delayed rewards were replayed successfully after recovery.

## Timeline

All times are in UTC.

| Time | Event |
| --- | --- |
| 18:02 | Backend version `2026.06.18.1` deployed to production. |
| 18:07 | First increase in matchmaking validation errors detected. |
| 18:11 | Support receives player reports of failed queue attempts and missing cosmetics. |
| 18:16 | On-call engineer acknowledges alert for elevated `matchmaking_start_failed` rate. |
| 18:22 | Initial investigation focuses on matchmaking queue workers. No queue backlog found. |
| 18:31 | Error logs show inventory validation failures during match session creation. |
| 18:44 | Redis cache inconsistency suspected after comparing cached inventory data with Postgres. |
| 18:53 | One backend instance found running previous build `2026.06.17.3`. |
| 18:58 | Stale instance drained from load balancer. |
| 19:06 | Redis keys for affected player inventory records invalidated. |
| 19:18 | Matchmaking success rate begins returning to normal. |
| 19:32 | Reward replay job started for delayed match rewards. |
| 19:49 | Metrics confirm normal matchmaking, inventory, and reward claim behavior. |
| 19:56 | Incident resolved. |

## Root Cause

The root cause was a partial backend deployment combined with incompatible cache behavior between two service versions.

Backend version `2026.06.18.1` introduced a new `inventory_revision` field used to detect whether cached inventory data was current. One production instance failed to restart during deployment because its process supervisor reported success before the old process had fully exited. As a result, the load balancer continued routing traffic to a stale instance.

The stale instance did not understand `inventory_revision`. When it handled inventory requests, it wrote older inventory payloads back into Redis without the new revision field. Newer matchmaking workers then rejected those cached records as invalid and failed match creation.

## Contributing Factors

- Deployment health checks verified HTTP availability but not the running application version.
- Cache payloads were not versioned.
- Matchmaking treated inventory validation failures as fatal instead of retrying against the database.
- The Redis invalidation runbook did not include player inventory keys.
- Alerting caught the symptom, but not the mixed-version deployment state.

## What Went Well

- On-call response began within five minutes of the first alert.
- No permanent player data loss occurred.
- Reward replay tooling worked as expected.
- Support and engineering used a shared incident channel, reducing duplicate investigation.

## What Went Poorly

- We spent too long investigating matchmaking workers before checking deployment consistency.
- The stale backend instance was not visible in the deployment dashboard.
- Players received generic matchmaking errors instead of a clearer service status message.
- Cache recovery required manual Redis key invalidation.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add application version checks to production health endpoints. | Backend | 2026-06-25 | Open |
| Block deployments if any instance reports an unexpected version. | Platform | 2026-06-28 | Open |
| Add schema versioning to Redis inventory payloads. | Backend | 2026-07-02 | Open |
| Update matchmaking to retry inventory reads from Postgres on cache validation failure. | Gameplay Services | 2026-07-05 | Open |
| Add mixed-version deployment alerting. | Platform | 2026-07-08 | Open |
| Expand Redis invalidation runbook to include inventory and reward keys. | Live Ops | 2026-06-24 | Open |
| Add player-facing degraded-service messaging for matchmaking failures. | Client | 2026-07-10 | Open |

## Follow-Up

We will review deployment safety checks before the next content patch. Until version-aware health checks are live, production deploys will require manual verification that all backend instances report the expected build version before the release is considered complete.