# Postmortem: Home Automation Hub Outage

## Summary

On 2026-07-08, the home-automation hub service experienced a partial outage that prevented devices from executing cloud-routed automations and remote commands. Local-only automations continued to run for most customers, but remote app control, voice-assistant integrations, and third-party webhook triggers were degraded or unavailable.

The outage lasted 1 hour and 42 minutes. Service was restored after rolling back a firmware coordination service deployment and draining stale hub sessions from the message broker.

## Impact

- 38% of active hubs were unable to receive remote commands.
- 24% of scheduled cloud-backed automations were delayed or skipped.
- Voice-assistant commands failed for affected users.
- Mobile app device state was stale for up to 90 minutes.
- Local automations configured to run entirely on the hub were not affected.

No customer data was lost.

## Timeline

All times are in UTC.

| Time | Event |
| --- | --- |
| 14:05 | Deployment of `hub-orchestrator` version `2026.07.08.1` begins. |
| 14:12 | Error rate on hub command delivery starts increasing. |
| 14:18 | Monitoring alerts on elevated MQTT reconnects and command timeouts. |
| 14:23 | On-call engineer begins investigation. |
| 14:31 | Customer support reports increased tickets for failed remote control and stale device status. |
| 14:39 | Initial hypothesis points to regional broker saturation. |
| 14:52 | Broker metrics show connection churn caused by hubs repeatedly reconnecting after rejected session handshakes. |
| 15:04 | Recent `hub-orchestrator` deployment identified as likely trigger. |
| 15:11 | Rollback initiated. |
| 15:24 | Rollback completes, but stale broker sessions continue blocking a subset of reconnecting hubs. |
| 15:37 | Broker session drain started for affected partitions. |
| 15:49 | Command delivery latency begins returning to normal. |
| 15:54 | Error rate drops below alert threshold. |
| 15:57 | Incident resolved. |
| 16:20 | Post-incident validation confirms remote commands, automations, and voice integrations are healthy. |

## Root Cause

A change in `hub-orchestrator` modified the hub session validation flow to require a newly issued capability token during reconnect. The deployment assumed all active hubs would already have the token cached from the previous heartbeat cycle.

That assumption was incorrect. Hubs running firmware versions older than `4.18.0` did not request the new capability token until after reconnect succeeded. As a result, those hubs entered a reconnect loop:

1. Hub reconnects to the broker.
2. Broker forwards session validation to `hub-orchestrator`.
3. `hub-orchestrator` rejects the session because the capability token is missing.
4. Hub disconnects and retries.
5. Stale broker sessions accumulate during rapid reconnect attempts.

The stale sessions amplified the issue by consuming broker partition capacity, which caused command delivery delays for some unaffected hubs as well.

## Contributing Factors

- The compatibility test suite did not include firmware versions older than `4.18.0`.
- The deployment was rolled out globally instead of by hub firmware cohort.
- Broker session cleanup was tuned for normal reconnect behavior, not reconnect storms.
- The new session validation path failed closed without a temporary compatibility fallback.

## Detection

The incident was detected by automated alerts for:

- Increased MQTT reconnect rate.
- Elevated command delivery timeout rate.
- Broker partition connection saturation.

Customer support reports confirmed the user-facing symptoms shortly after the first alert.

## Resolution

We restored service by:

- Rolling back `hub-orchestrator` to the previous stable version.
- Draining stale broker sessions from affected partitions.
- Temporarily disabling strict capability-token enforcement for hub reconnects.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add compatibility tests for all supported hub firmware versions. | Platform | 2026-07-15 | Open |
| Change session validation rollout to support firmware-based canaries. | Hub Services | 2026-07-19 | Open |
| Add reconnect storm simulation to broker load testing. | Infrastructure | 2026-07-22 | Open |
| Implement broker-side stale session cleanup for rapid reconnect loops. | Infrastructure | 2026-07-26 | Open |
| Add deployment guardrail for reconnect-rate regression. | SRE | 2026-07-18 | Open |
| Document capability-token migration requirements for firmware and cloud teams. | Platform | 2026-07-16 | Open |

## What Went Well

- Monitoring detected the incident quickly.
- Rollback completed without additional service disruption.
- Local-only automations continued working for most affected households.

## What Could Be Improved

- Compatibility coverage needs to reflect the actual firmware distribution in the field.
- Global rollout was too broad for a session validation change.
- Broker recovery required manual intervention and should be automated.