# Postmortem: Home-Automation Hub Control Outage

**Incident date:** 2026-07-08  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

A firmware update caused a subset of home-automation hubs to lose local device control and cloud connectivity after reboot. Affected customers were unable to control lights, locks, thermostats, and scenes through the mobile app or voice assistants. Some scheduled automations continued to run locally, but event-triggered automations were delayed or skipped.

## Customer Impact

- Approximately 18% of active hubs were affected.
- Mobile app control failed or showed devices as offline.
- Voice assistant integrations returned timeout errors.
- Event-based automations, such as motion-triggered lights, were delayed or missed.
- No evidence of device state corruption, unauthorized access, or data loss was found.

## Timeline

All times in UTC.

| Time | Event |
|---|---|
| 14:00 | Firmware version `4.18.2` rollout begins to 10% of production hubs. |
| 14:17 | First increase in hub reconnect attempts detected, below alert threshold. |
| 14:29 | Customer support receives first reports of offline hubs. |
| 14:36 | Cloud connectivity error rate exceeds alert threshold. |
| 14:39 | On-call engineer acknowledges SEV-1 alert. |
| 14:46 | Rollout of firmware `4.18.2` is paused. |
| 15:02 | Logs show affected hubs repeatedly restarting the local automation service. |
| 15:18 | Engineering identifies a migration failure in the local device registry cache. |
| 15:31 | Hotfix firmware `4.18.3` is built with migration rollback protection. |
| 15:49 | Hotfix rollout begins for affected hubs. |
| 16:08 | Hub reconnect rates begin returning to baseline. |
| 16:22 | Error rates return to normal levels. |
| 16:35 | Incident declared resolved. |

## Root Cause

Firmware version `4.18.2` included a migration that changed the format of the local device registry cache. The migration assumed that all cached device records contained a non-null `protocol` field.

Older hubs that had paired Zigbee devices before firmware `4.5.0` could still contain records without this field. When those hubs rebooted after the update, the migration failed, leaving the registry cache partially written.

The local automation service then attempted to load the incomplete cache, crashed, and restarted repeatedly. Because the same service also owns the hub's cloud session heartbeat, affected hubs appeared offline and could not reliably process commands.

## Contributing Factors

- The migration test dataset did not include legacy device records from pre-`4.5.0` hubs.
- The local automation service and cloud heartbeat shared a process, increasing blast radius.
- Rollout health checks focused on firmware installation success, not post-reboot service stability.
- Customer support reports arrived before internal alerting reached SEV-1 thresholds.

## Resolution

We paused the firmware rollout, built hotfix `4.18.3`, and deployed it to affected hubs. The hotfix validates registry records before migration, fills missing `protocol` values from paired-device metadata, and restores from the previous cache snapshot if migration fails.

## Action Items

| Action | Owner | Due Date | Status |
|---|---|---:|---|
| Add legacy registry fixtures to firmware migration tests. | Firmware Team | 2026-07-15 | Open |
| Split cloud heartbeat from the local automation service process. | Hub Platform Team | 2026-08-02 | Open |
| Add post-reboot crash-loop detection to rollout gates. | Release Engineering | 2026-07-19 | Open |
| Lower alert threshold for hub reconnect spikes during staged rollouts. | SRE | 2026-07-12 | Open |
| Add automatic rollback when service restarts exceed threshold after update. | Firmware Team | 2026-07-26 | Open |
| Update customer support runbook for firmware-induced hub offline incidents. | Support Ops | 2026-07-16 | Open |

## What Went Well

- The staged rollout limited the outage to a minority of hubs.
- The previous registry cache snapshot allowed recovery without factory resets.
- Hotfix deployment succeeded without requiring customer action.

## What Could Improve

- Migration tests need to reflect real long-lived hub data.
- Rollout gates need to measure service health after reboot, not just install success.
- Critical hub functions should be isolated so one service crash does not break both local control and cloud connectivity.