# Postmortem: Matchmaking and Login Outage

**Studio:** Moonbyte Games  
**Game:** *Starfall Kitchen*  
**Date:** 2026-07-08  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 8, players were unable to log in or enter multiplayer matchmaking for *Starfall Kitchen*. The outage began after a backend deployment introduced a database connection leak in the session service. As traffic increased after the daily challenge reset, the service exhausted available database connections, causing login requests and matchmaking ticket creation to fail.

## Timeline

All times in UTC.

| Time | Event |
|---|---|
| 16:00 | Daily challenge reset triggered normal player traffic spike. |
| 16:08 | Error rate on `/session/start` increased above 20%. |
| 16:12 | First player reports appeared in Discord and Steam forums. |
| 16:17 | On-call engineer received alert for elevated 5xx responses. |
| 16:24 | Initial investigation focused on matchmaking service health. |
| 16:39 | Engineers identified database connection pool saturation. |
| 16:47 | Session service pods were restarted, temporarily restoring logins. |
| 17:03 | Error rate climbed again as new connections leaked under load. |
| 17:21 | Recent deployment was identified as the likely source. |
| 17:35 | Backend deployment was rolled back to the previous version. |
| 17:49 | Database connections stabilized. |
| 18:02 | Login and matchmaking success rates returned to normal. |
| 18:25 | Incident was declared resolved after sustained recovery. |

## Customer Impact

Approximately 38% of active players were affected during the incident window.

Players experienced:

- Failed login attempts
- Infinite loading screens after pressing "Play Online"
- Failed matchmaking ticket creation
- Party disbands when one member could not establish a session
- Lost access to the daily challenge during part of its peak window

No player inventory, currency, save data, or progression data was lost.

## Root Cause

A change in the session service introduced a new code path for refreshing player entitlement data during login. The new path opened a database transaction but did not reliably close it when the entitlement provider returned a timeout.

During normal traffic, the leak was slow enough to avoid immediate detection. After the daily challenge reset, concurrent login volume increased sharply. The leaked transactions accumulated until the database connection pool was exhausted.

Once the pool was exhausted, healthy services that depended on the same database could not acquire connections. This caused cascading failures in login, party validation, and matchmaking ticket creation.

## Contributing Factors

- The new entitlement refresh path was not covered by load testing.
- Database pool saturation alerts were configured for production databases but not for individual service pools.
- The deployment occurred two hours before the daily challenge traffic spike.
- The session service retry behavior amplified pressure on the database during partial failures.
- The rollback process required manual verification because the deployment pipeline does not currently support one-click rollback.

## Resolution

The team rolled back the session service to the previous stable version. After rollback, leaked transactions stopped accumulating and database connection usage returned to normal. Session service pods were restarted to clear stuck connections.

A hotfix was later prepared to ensure all transaction paths use scoped cleanup and timeout-safe error handling.

## What Went Well

- Player reports in Discord helped confirm the issue quickly.
- Restarting the session service provided temporary relief while investigation continued.
- Rollback resolved the issue without data repair.
- The team maintained updates in Discord, Steam forums, and the status page throughout the incident.

## What Went Poorly

- Monitoring identified elevated errors but did not point directly to connection pool exhaustion.
- The team initially investigated matchmaking rather than the session service.
- The problematic deployment passed staging because staging traffic does not resemble reset-time production load.
- Status page updates lagged behind Discord updates during the first 30 minutes.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add service-level database pool saturation alerts | Backend | High | 2026-07-12 |
| Add load test covering daily challenge reset traffic | Platform | High | 2026-07-15 |
| Refactor session service database access to use scoped transaction helpers | Backend | High | 2026-07-16 |
| Add automated rollback support to deployment pipeline | DevOps | Medium | 2026-07-22 |
| Add entitlement provider timeout tests | Backend | Medium | 2026-07-18 |
| Update incident runbook with database pool diagnostics | Engineering | Medium | 2026-07-14 |
| Create status update checklist for Discord, Steam, and status page | Community | Low | 2026-07-13 |

## Prevention

Future backend deployments will avoid the two-hour window before daily reset unless they are emergency fixes. Any change touching login, sessions, entitlements, or matchmaking will require load testing against reset-like traffic before production release.