# Postmortem: Multiplayer Login and Matchmaking Outage

**Studio:** Northstar Finch Games  
**Game:** *Skyline Salvage*  
**Date of incident:** 2026-07-11  
**Duration:** 2 hours 37 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On 2026-07-11, players were unable to reliably log in, form parties, or enter multiplayer sessions for *Skyline Salvage*. The outage began shortly after we deployed backend version `2026.07.11.2`, which introduced a new entitlement reconciliation step during player login.

The new code path made an unindexed database query for every login attempt. Under normal weekend traffic, this saturated the primary PostgreSQL database, exhausted API connection pools, and caused cascading failures in authentication and matchmaking.

We mitigated the incident by rolling back the release, temporarily disabling entitlement reconciliation, and clearing overloaded worker queues. Service recovered at 20:44 UTC.

## Customer Impact

Between 18:07 and 20:44 UTC:

- Approximately 41,000 login attempts failed or timed out.
- Matchmaking success rate dropped from 96% to 18%.
- Existing multiplayer sessions were mostly unaffected, but party creation and requeueing failed frequently.
- Players who purchased DLC during the incident saw delayed entitlement grants.
- We received 612 support tickets and roughly 1,900 Discord reports.
- No player inventory, save data, or purchase records were lost.

## Timeline

All times are UTC.

| Time | Event |
| --- | --- |
| 17:52 | Backend version `2026.07.11.2` deployed to production. |
| 18:07 | Login latency begins increasing above normal weekend baseline. |
| 18:13 | First Discord reports appear: players stuck on "Connecting to services." |
| 18:18 | On-call engineer receives alert for elevated API 5xx rate. |
| 18:24 | Matchmaking queue depth exceeds alert threshold. |
| 18:31 | Initial investigation focuses on matchmaking workers due to queue growth. |
| 18:46 | Database CPU reaches 98%; connection pool saturation observed. |
| 18:58 | Team identifies slow entitlement query introduced in latest deploy. |
| 19:06 | Rollback started for backend version `2026.07.11.2`. |
| 19:19 | Rollback completes, but login errors continue due to saturated workers and retry backlog. |
| 19:31 | Entitlement reconciliation disabled via feature flag. |
| 19:44 | API error rate starts dropping. |
| 20:08 | Matchmaking workers restarted in batches to drain stuck jobs. |
| 20:31 | Login success rate returns above 95%. |
| 20:44 | Incident declared resolved. |
| 21:10 | Public status page and Discord update posted with refund/entitlement delay guidance. |

## Root Cause

The root cause was an inefficient database query introduced in backend version `2026.07.11.2`.

The release added entitlement reconciliation during login so players could receive newly purchased DLC immediately after authentication. The query searched purchase records using `LOWER(email)` and `platform_account_id`, but the production database did not have a matching index for that access pattern.

As login traffic increased during peak weekend hours, PostgreSQL performed repeated sequential scans across the purchase table. This caused:

1. High database CPU usage.
2. Slow authentication responses.
3. API connection pool exhaustion.
4. Client retries that amplified load.
5. Matchmaking failures because matchmaking requires a valid session token from the auth service.

The migration adding the intended index existed in the branch but was not included in the production migration bundle due to a packaging mistake in our release script.

## Contributing Factors

- The new login path was tested with staging data that was too small to expose the slow query.
- The entitlement feature shipped enabled for 100% of players instead of behind a gradual rollout.
- Our alerting detected API failures, but not the underlying slow query quickly enough.
- Matchmaking had a hard dependency on synchronous auth validation during queue entry.
- The rollback did not immediately recover service because retry queues and worker pools were already saturated.

## What Went Well

- The team identified the bad deploy within one hour.
- Feature flags allowed us to disable entitlement reconciliation without another deploy.
- No persistent player data was lost.
- Community and support teams coordinated updates quickly once the incident was confirmed.

## What Went Wrong

- The release process allowed application code and database migrations to get out of sync.
- We did not load test the login path with production-scale purchase data.
- The deploy was not rolled out gradually.
- Initial triage focused on matchmaking symptoms instead of shared dependencies.

## Action Items

| Action Item | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Add production-equivalent data volume tests for login and entitlement queries. | Backend | High | 2026-07-25 |
| Update release pipeline to fail if expected migrations are missing from the bundle. | DevOps | High | 2026-07-22 |
| Require gradual rollout for all login-path changes. | Engineering Lead | High | 2026-07-19 |
| Add slow-query alerts for auth and entitlement database calls. | Backend | Medium | 2026-07-26 |
| Add circuit breaker around entitlement reconciliation during login. | Backend | Medium | 2026-08-02 |
| Decouple matchmaking queue entry from synchronous entitlement checks. | Gameplay Services | Medium | 2026-08-09 |
| Increase staging purchase dataset size to match production query characteristics. | QA / Backend | Medium | 2026-07-29 |
| Write incident runbook for database saturation affecting auth. | SRE | Medium | 2026-07-31 |

## Prevention

Future changes to authentication, purchases, entitlements, or matchmaking entry will require:

- A staged rollout plan.
- Query plans reviewed against production-like data.
- Migration verification in CI.
- A documented rollback path.
- Feature flags for any non-critical login work.

## Customer Follow-Up

We posted a status update on Discord, Steam, and our public status page. Players who purchased DLC during the outage had their entitlements reconciled after recovery. As a goodwill gesture, all players who logged in within 24 hours of the incident received 500 scrap credits.