# Postmortem: Home Automation Hub Outage

**Date:** 2026-07-08  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Services affected:** Hub command routing, mobile app device control, automation execution

## Summary

On July 8, 2026, a backend deployment caused a subset of home-automation hubs to lose connectivity with the command routing service. Customers were unable to control devices through the mobile app, voice assistants, and scheduled automations. Local manual control of devices continued to work where supported.

The outage was caused by a schema change in the hub session registry that was deployed before all command-routing workers had been updated to read the new format. Older workers treated valid hub sessions as expired and repeatedly disconnected active hubs.

## Customer Impact

Approximately 18% of active hubs were affected.

Affected customers experienced:

- Delayed or failed device commands from the mobile app
- Missed scheduled automations, including lighting and thermostat routines
- Failed voice-assistant commands
- Incorrect “hub offline” status in the app
- Repeated reconnect events on affected hubs

No customer device configurations, automation rules, or account data were lost.

## Timeline

All times are in UTC.

| Time | Event |
| --- | --- |
| 14:02 | Deployment of session registry migration begins. |
| 14:08 | New registry writer starts storing hub session records using `expires_at_ms` instead of `ttl_seconds`. |
| 14:11 | Error rate begins increasing on command-routing workers still running the previous version. |
| 14:16 | Monitoring detects elevated hub disconnects and command failures. |
| 14:19 | Incident declared as SEV-2. |
| 14:25 | On-call engineer identifies a spike in `SESSION_EXPIRED` responses. |
| 14:37 | Rollback attempted for the registry writer, but mixed-format records remain in cache. |
| 14:49 | Command-routing deployment paused. |
| 15:03 | Compatibility patch prepared to read both session formats. |
| 15:21 | Patch deployed to all command-routing workers. |
| 15:32 | Hub reconnect rate returns to baseline. |
| 15:49 | Command success rate fully recovers. |
| 15:56 | Incident resolved. |

## Root Cause

The root cause was an incompatible session record format introduced by the hub session registry migration.

The migration changed session expiration storage from:

```json
{
  "hub_id": "hub_123",
  "ttl_seconds": 300
}
```

to:

```json
{
  "hub_id": "hub_123",
  "expires_at_ms": 1783520100000
}
```

The new writer was deployed before all command-routing workers were updated. Older workers did not recognize `expires_at_ms`, defaulted missing `ttl_seconds` values to zero, and marked active sessions as expired.

This caused hubs to be disconnected even though they were healthy and actively heartbeating.

## Contributing Factors

- The migration plan assumed writer and reader deployments would complete within the same rollout window.
- Backward compatibility was tested in staging, but staging did not include mixed-version workers.
- The deployment checklist did not require validating reader compatibility before enabling the new writer.
- Alerting detected disconnect volume, but there was no alert specifically tied to session parsing failures.

## Resolution

We deployed a compatibility patch to command-routing workers so they could read both `ttl_seconds` and `expires_at_ms`.

We also flushed expired session-cache entries created by the old workers and allowed affected hubs to reconnect normally. No customer action was required.

## What Went Well

- Existing disconnect-rate alerts detected the issue within minutes.
- The incident commander paused further rollout quickly.
- The compatibility patch was small and low risk.
- Hub reconnect behavior recovered without requiring device resets.

## What Went Wrong

- The schema migration was not safely backward compatible.
- Mixed-version deployment behavior was not represented in staging.
- Rollback did not immediately resolve the issue because incompatible records were already cached.
- Customer support did not receive an incident brief until 38 minutes after declaration.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Require backward-compatible readers before session schema writer changes | Platform | 2026-07-15 | Open |
| Add mixed-version deployment tests for command-routing services | QA | 2026-07-19 | Open |
| Add alert for session parsing fallback/default behavior | Observability | 2026-07-17 | Open |
| Update deployment checklist for schema migrations | Release Engineering | 2026-07-16 | Open |
| Reduce session-cache TTL during migrations | Platform | 2026-07-22 | Open |
| Create customer-support incident briefing template | Support Operations | 2026-07-18 | Open |

## Prevention

Future hub-session schema changes must follow an expand-and-contract migration:

1. Deploy readers that support both old and new formats.
2. Verify all readers are upgraded.
3. Enable writers for the new format.
4. Monitor mixed-format behavior.
5. Remove old-format support only after the migration window closes.

This process will be added to the release checklist for all hub connectivity and automation-routing services.