# Postmortem: Matchmaking and Login Outage

Date: 2026-07-08  
Studio: Ember North Games  
Game: Starfall Courier  
Status: Final  
Severity: SEV-1  
Duration: 2 hours 17 minutes

## Summary

On July 8, 2026, Starfall Courier players were unable to log in, create parties, or enter matchmaking for 2 hours and 17 minutes. The outage began after a backend deployment that introduced a new Redis key format for session tokens. The change was incompatible with an older login service instance that remained in rotation due to a failed rolling restart.

The incident was resolved by draining the stale service instance, rolling back the session-token change, and clearing malformed session entries from Redis.

## Customer Impact

- Approximately 41,000 players attempted to log in during the outage window.
- 73% of login attempts failed with `SESSION_INVALID`.
- Players already in active matches were mostly unaffected.
- Players returning to the main menu were disconnected and unable to rejoin matchmaking.
- Party invites failed or expired immediately.
- The Steam review score for the day dropped from 86% positive to 62% positive.
- Player support received 1,184 tickets and 3,900 Discord reports.

## Timeline

All times are UTC.

| Time | Event |
| --- | --- |
| 18:03 | Backend version `2026.07.08.1` deployed to production. |
| 18:07 | Login error rate increased above normal baseline. |
| 18:10 | Discord moderators reported widespread login failures in `#server-status`. |
| 18:14 | On-call engineer acknowledged alerts for elevated `SESSION_INVALID` responses. |
| 18:21 | Initial hypothesis focused on Steam auth provider instability. No provider outage was found. |
| 18:32 | Matchmaking queue depth dropped by 68%, confirming impact beyond new logins. |
| 18:41 | Engineers identified inconsistent session-token formats in Redis. |
| 18:54 | Deployment logs showed one old login-service instance still receiving traffic. |
| 19:02 | Stale instance was drained from the load balancer. Error rate decreased but did not recover fully. |
| 19:18 | Team discovered malformed session entries created during mixed-version operation. |
| 19:27 | Backend deployment was rolled back to `2026.07.07.3`. |
| 19:46 | Redis cleanup job started for malformed session keys. |
| 20:08 | Login success rate returned to normal. |
| 20:20 | Matchmaking and party services were confirmed stable. |
| 20:31 | Incident marked resolved. |

## Root Cause

The outage was caused by an incompatible session-token format introduced in the login service.

The new deployment changed session keys from:

`session:{player_id}`

to:

`session:v2:{platform}:{player_id}`

The updated login service could read both formats, but one older login-service instance remained active after a rolling restart failed silently. That older instance could only read and write the original format.

Because both service versions were running at the same time, players were issued session records in inconsistent formats. Downstream services, including party and matchmaking, rejected some of these sessions as invalid. The issue persisted after the stale instance was drained because malformed session entries remained in Redis until they were manually cleaned up.

## Contributing Factors

- The rolling deployment reported success even though one instance failed to restart.
- The session-token migration was not protected by a feature flag.
- Compatibility testing covered old clients with new services, but not mixed backend versions.
- Redis session keys had no schema version validation at write time.
- The login alert fired correctly, but the runbook initially pointed engineers toward external auth providers.

## What Went Well

- Player reports in Discord helped confirm impact quickly.
- Existing dashboards made it clear that matchmaking failures were tied to login/session validation.
- Rollback completed cleanly once the bad deployment was identified.
- No player inventory, currency, or progression data was lost.

## What Went Wrong

- The deployment system allowed a partial rollout to remain in production.
- The new session format was released without a staged migration plan.
- Engineers spent the first 25 minutes investigating the wrong dependency.
- Support and community teams did not receive a clear status update until 46 minutes into the incident.

## Detection

The incident was detected by automated alerts for elevated login failures and confirmed by player reports. The first alert fired 7 minutes after deployment.

## Resolution

The team resolved the incident by:

1. Draining the stale login-service instance from production traffic.
2. Rolling back the backend deployment to the previous stable version.
3. Running a Redis cleanup job to remove malformed session keys.
4. Verifying login, party creation, and matchmaking success rates across Steam, PlayStation, and Xbox.

## Action Items

| Action Item | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Make deployment health checks fail if any instance remains on the previous version. | Platform | P0 | 2026-07-15 |
| Add mixed-version compatibility tests for backend services. | Backend | P0 | 2026-07-19 |
| Require feature flags for session, auth, and matchmaking schema changes. | Backend | P0 | 2026-07-22 |
| Add Redis session schema validation and metrics by schema version. | Backend | P1 | 2026-07-26 |
| Update login outage runbook to include backend version skew checks. | SRE | P1 | 2026-07-17 |
| Add automated cleanup tooling for invalid session records. | Platform | P1 | 2026-07-29 |
| Create a community incident template for faster status updates. | Community | P2 | 2026-07-16 |

## Preventative Measures

Future session-format changes will use a staged migration:

1. Deploy readers that support both old and new formats.
2. Enable dual writes behind a feature flag.
3. Validate metrics for both formats.
4. Switch reads to the new format.
5. Remove old-format support only after expiration of all old sessions.

No auth or matchmaking schema change will be deployed without mixed-version testing and explicit rollback instructions.