# Postmortem: Matchmaking and Account Service Outage

**Date of incident:** 2026-06-28  
**Duration:** 2 hours 17 minutes  
**Services affected:** Matchmaking, player profiles, inventory sync, login sessions  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On June 28, 2026, players of *Moonfall Circuit* were unable to reliably log in, join matches, or receive post-match rewards for just over two hours. The outage began shortly after a backend deployment that introduced a new player-presence write path. Under live traffic, the change generated significantly more database writes than expected, saturating the primary PostgreSQL instance and causing cascading timeouts across dependent services.

We restored service by rolling back the deployment, disabling presence writes through a feature flag, and manually clearing a backlog of stalled reward-processing jobs.

## Customer Impact

Approximately 18,400 players were affected during the incident window.

Impact included:

- 71% of login attempts failed or timed out.
- Matchmaking queue times increased from a median of 18 seconds to over 6 minutes.
- Completed matches did not consistently grant XP, currency, or event progress.
- Some players were disconnected at match end when profile updates timed out.
- No player inventory data was permanently lost.

We replayed delayed reward jobs after recovery. Players who completed matches during the outage received earned rewards by 22:40 UTC.

## Timeline

All times are UTC.

| Time | Event |
| --- | --- |
| 18:03 | Backend version `2026.06.28.3` deployed to production. |
| 18:09 | Database CPU rises above 85%; connection pool saturation begins. |
| 18:12 | First elevated login timeout alert fires. |
| 18:16 | Player support reports a spike in Discord and support ticket complaints. |
| 18:21 | On-call engineer declares SEV-1 and joins incident channel. |
| 18:28 | Matchmaking error rate reaches 54%; deployment is suspected but not yet confirmed. |
| 18:36 | Database logs show heavy write volume from the new presence update endpoint. |
| 18:42 | Feature flag for presence writes is disabled, but existing workers continue retrying queued writes. |
| 18:51 | Rollback of backend version `2026.06.28.3` begins. |
| 19:07 | Rollback completes; login success rate begins recovering. |
| 19:18 | Reward-processing queue is paused to reduce database pressure. |
| 19:34 | Database CPU returns below 60%; matchmaking stabilizes. |
| 19:51 | Reward-processing queue is resumed with reduced concurrency. |
| 20:20 | Login, matchmaking, and profile APIs return to normal error rates. |
| 20:40 | Incident marked resolved. |
| 22:40 | Delayed match rewards fully replayed. |

## Root Cause

The outage was caused by an unbounded increase in writes to the player-presence table introduced in backend version `2026.06.28.3`.

The new presence system was intended to update a player's online status when they entered matchmaking, joined a lobby, started a match, completed a match, or returned to the main menu. In testing, this produced a small number of writes per session. In production, the game client retried state transitions more frequently during normal packet loss and scene-loading delays, resulting in repeated presence updates for the same player.

The database table had an index on `updated_at` used by internal moderation tooling. During peak traffic, the repeated updates caused excessive index churn, high write amplification, and connection pool exhaustion. Once the profile service could not acquire database connections, login, matchmaking, inventory sync, and reward finalization all began timing out.

## Contributing Factors

- Load testing did not simulate client retry behavior during scene transitions.
- The presence feature flag disabled new writes at the API layer but did not stop queued worker retries.
- Match reward finalization shared the same database pool as login and profile reads.
- The deployment happened during a weekend event with higher-than-normal concurrency.
- The dashboard showed API latency clearly but did not immediately identify database write amplification.

## What Went Well

- The on-call engineer declared SEV-1 quickly after confirming player-facing impact.
- Rollback completed cleanly without schema changes.
- Reward jobs were idempotent, allowing delayed rewards to be safely replayed.
- Community and support teams posted status updates within 30 minutes.

## What Went Wrong

- The risky write-path change was deployed during a live event.
- The feature flag did not fully disable all execution paths.
- Database capacity alerts fired after players were already seeing failures.
- The incident team initially focused on matchmaking instead of the shared profile database dependency.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add production-like retry behavior to backend load tests. | Gameplay Services | 2026-07-12 | Open |
| Move presence writes to Redis with periodic database compaction. | Backend | 2026-07-19 | Open |
| Split login/profile reads and reward writes into separate database pools. | Platform | 2026-07-16 | Open |
| Add a kill switch that stops API writes and worker retries for flagged features. | Backend | 2026-07-14 | Open |
| Block non-critical backend deployments during live events. | Engineering Management | 2026-07-11 | Open |
| Add alerts for write amplification and index update volume. | Infrastructure | 2026-07-13 | Open |
| Document incident roles and escalation steps for weekend coverage. | Production | 2026-07-15 | Open |

## Prevention

We will not redeploy the presence system until it has been moved off the primary relational write path, covered by load tests that include client retry behavior, and protected by a full kill switch. Future live-event periods will require a deployment freeze for non-critical backend changes unless explicitly approved by the incident commander and engineering lead.