# Postmortem: Home-Automation Hub Outage

**Incident Date:** 2026-06-18  
**Duration:** 1 hour 42 minutes  
**Severity:** SEV-1  
**Status:** Closed

## Summary

On June 18, a backend configuration rollout caused a large portion of home-automation hubs to lose connectivity with the control plane. Affected hubs remained powered and continued running local automations, but users could not reliably control devices through the mobile app, receive live device state updates, or use cloud-dependent integrations.

The issue was caused by an invalid MQTT broker routing configuration that was accepted by our deployment pipeline but rejected by hub clients after reconnect. We rolled back the configuration and restarted the affected broker pool to clear stale sessions.

## Customer Impact

Approximately 38% of active hubs were affected.

Affected customers experienced:

- Mobile app commands timing out or showing stale device state
- Delayed or missing notifications for sensors, locks, and alarms
- Voice assistant integrations failing for cloud-routed commands
- New device pairing failures during the incident window

Unaffected functionality:

- Local automations continued running on the hub
- Existing Zigbee, Z-Wave, Matter, and LAN device connections remained active
- Manual device controls, such as wall switches and keypads, continued working

## Timeline

All times are in UTC.

| Time | Event |
| --- | --- |
| 14:02 | Configuration rollout began for MQTT broker routing changes. |
| 14:07 | First increase in hub reconnect attempts observed. |
| 14:11 | Mobile app command latency exceeded alert threshold. |
| 14:14 | On-call engineer paged for elevated command timeout rate. |
| 14:19 | Support began receiving customer reports of offline hubs. |
| 14:26 | Incident declared SEV-1. |
| 14:31 | Initial investigation focused on broker capacity due to reconnect surge. |
| 14:44 | Broker metrics showed healthy CPU and memory but abnormal session rejection rates. |
| 14:53 | Recent routing config rollout identified as likely trigger. |
| 15:01 | Rollback initiated. |
| 15:13 | Rollback completed, but some hubs remained disconnected due to stale broker sessions. |
| 15:22 | Affected broker pool restarted in batches. |
| 15:39 | Hub reconnect rate returned to normal. |
| 15:44 | Mobile app command success rate recovered above 99%. |
| 15:49 | Incident marked resolved. |

## Root Cause

The outage was caused by an invalid broker routing configuration deployed to production.

The new configuration introduced a region alias, `na-east-primary`, for hub MQTT routing. The backend accepted the alias, but the embedded hub client only accepted region identifiers from a fixed allowlist. When affected hubs reconnected, they received the new alias, rejected the routing response, and retried connection setup. This created a reconnect loop.

The deployment pipeline validated the configuration schema but did not validate compatibility with currently deployed hub firmware versions. As a result, the change passed automated checks even though roughly 38% of production hubs were running firmware that could not accept the new alias format.

## Contributing Factors

- Firmware compatibility validation was not included in the config rollout pipeline.
- The rollout targeted all North American broker routes at once instead of using a staged canary.
- Broker dashboards emphasized infrastructure saturation, which delayed identification of client-side session rejection.
- The mobile app showed affected hubs as offline without distinguishing cloud connectivity loss from local hub failure.

## Resolution

We resolved the incident by rolling back the routing configuration to the previous region identifiers and restarting the affected broker pool in batches to clear stale sessions. Hubs reconnected normally after receiving compatible routing responses.

## What Went Well

- Local automations continued running, reducing safety and convenience impact.
- Existing broker health dashboards confirmed that the issue was not infrastructure capacity.
- Rollback was available and completed quickly once the configuration was identified.
- Support and engineering coordinated customer messaging during the incident.

## What Did Not Go Well

- The invalid configuration reached production.
- The first alert pointed to app command latency rather than hub routing failures.
- The canary process was bypassed because the change was classified as configuration-only.
- Customer-facing status updates lagged behind internal incident declaration by 18 minutes.

## Action Items

| Action | Owner | Due Date | Status |
| --- | --- | --- | --- |
| Add firmware compatibility checks to broker routing config validation. | Platform | 2026-07-02 | Open |
| Require staged rollout for all hub connectivity configuration changes. | SRE | 2026-07-05 | Open |
| Add alert for elevated MQTT session rejection by reason code. | Observability | 2026-06-28 | In Progress |
| Update mobile app offline messaging to distinguish cloud connectivity issues from local hub failure. | Mobile | 2026-07-12 | Open |
| Add runbook section for reconnect-loop diagnosis. | SRE | 2026-06-25 | Open |
| Review configuration-only deployment policy and approval requirements. | Engineering Leadership | 2026-07-08 | Open |

## Preventative Measures

Going forward, hub connectivity configuration changes will follow the same rollout controls as service deployments. Any change that affects connection setup, region routing, authentication, or broker selection must pass compatibility validation against supported firmware versions and must begin with a limited canary before broader rollout.