# Postmortem: Home Automation Hub Control Plane Outage

**Date of incident:** 2026-07-09  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Authors:** Platform Engineering, Hub Runtime Team

## Summary

On July 9, 2026, a bad configuration rollout caused a large percentage of home-automation hubs to lose connectivity to the cloud control plane. During the outage, affected customers were unable to control devices through the mobile app, run cloud-triggered automations, or receive remote device state updates.

Local automations and direct device controls continued to work for hubs that had already cached rules and device credentials. However, cloud-dependent features such as voice assistant integrations, remote access, push notifications, and scene execution from the app were degraded or unavailable.

The incident was caused by an invalid MQTT broker routing configuration deployed to production. The configuration passed schema validation but mapped a subset of hub regions to an empty broker pool. Hubs in those regions repeatedly attempted to reconnect, causing elevated load on authentication and presence services and delaying recovery after rollback.

## Customer Impact

Approximately 38% of active hubs were affected.

Affected customers experienced one or more of the following:

- Mobile app showed hubs as offline.
- Remote device control failed or timed out.
- Cloud-triggered automations did not execute.
- Voice assistant commands failed.
- Push notifications for locks, alarms, cameras, and sensors were delayed or missing.
- Device state shown in the app was stale.

Safety-critical local routines, including smoke alarm relay, water shutoff automations, and scheduled local lighting rules, continued to run on hubs where those automations had previously been synced.

No customer device credentials, personal data, video streams, or automation definitions were lost.

## Timeline

All times are in UTC.

| Time | Event |
| --- | --- |
| 14:02 | Routing configuration version `mqtt-routing-2026.07.09.3` deployed to production. |
| 14:04 | Hub reconnect rate begins rising in `us-east`, `us-central`, and `eu-west`. |
| 14:07 | Presence service error rate exceeds alert threshold. On-call engineer paged. |
| 14:10 | Customer support begins receiving reports of hubs appearing offline in the mobile app. |
| 14:13 | Initial investigation focuses on presence service due to elevated 5xx responses. |
| 14:20 | Platform team identifies a sharp increase in MQTT connection failures with reason `NO_BROKER_AVAILABLE`. |
| 14:27 | Incident declared SEV-1. Engineering, support, and communications leads join bridge. |
| 14:33 | Recent routing configuration rollout identified as likely trigger. |
| 14:39 | Rollback initiated to `mqtt-routing-2026.07.09.2`. |
| 14:46 | Broker assignment errors begin decreasing, but reconnect storm continues to overload authentication service. |
| 14:52 | Rate limits temporarily relaxed for hub token refresh endpoints. |
| 15:01 | MQTT connection success rate returns above 90%. |
| 15:18 | Most hubs reconnect successfully; app hub-online state begins recovering. |
| 15:34 | Cloud automation execution backlog drains. |
| 15:49 | Metrics return to normal operating range. |
| 15:55 | Incident resolved. |
| 17:10 | Customer-facing incident summary published. |

## Root Cause

The outage was caused by an invalid production routing configuration for MQTT broker assignment.

The routing config introduced a new region-to-pool mapping format intended to support capacity-aware routing. The config validator checked that each referenced region and pool name was syntactically valid, but it did not verify that every region resolved to at least one active broker.

For three production regions, the deployed config matched hubs into a pool selector that returned an empty broker list. Affected hubs then entered reconnect loops. Each reconnect attempt refreshed auth state and updated hub presence, which increased load on shared services and amplified the incident.

The rollback corrected the routing map, but recovery was slowed by synchronized reconnect behavior across many hubs.

## Contributing Factors

- Config validation did not include semantic checks against live broker inventory.
- The rollout was applied globally instead of progressively by region.
- Hub reconnect backoff had insufficient jitter for large regional failures.
- Alerting fired on downstream presence errors before the more specific broker-assignment failure was surfaced.
- Runbooks did not clearly identify MQTT routing config as a common dependency for presence, remote control, and cloud automation failures.

## What Went Well

- Local hub automations continued to operate for most affected customers.
- Rollback completed quickly once the routing config was identified.
- Engineering, support, and communications teams coordinated through a single incident channel.
- No customer data was lost or corrupted.

## What Went Poorly

- Detection pointed responders toward a symptom service rather than the failing dependency.
- Recovery was delayed by reconnect storms after rollback.
- Customer support received reports before the incident was visible on the public status page.
- The config deployment process allowed a global production change without staged validation.

## Action Items

| Owner | Action Item | Due Date | Status |
| --- | --- | --- | --- |
| Platform Engineering | Add semantic validation requiring every routing rule to resolve to at least one healthy broker before deploy. | 2026-07-16 | Open |
| Platform Engineering | Change MQTT routing config rollout to staged regional deployment with automatic rollback on connection failure spikes. | 2026-07-23 | Open |
| Hub Runtime Team | Increase reconnect jitter and add adaptive backoff when broker assignment fails. | 2026-07-30 | Open |
| SRE | Add alert for `NO_BROKER_AVAILABLE` errors with routing config version included in the alert payload. | 2026-07-18 | Open |
| SRE | Update incident runbook for hub-offline events to include MQTT routing and broker inventory checks. | 2026-07-19 | Open |
| Customer Support | Add support macro for partial hub outages explaining local automation behavior and cloud feature impact. | 2026-07-15 | Open |
| Release Engineering | Require approval from Platform and SRE for global control-plane config changes. | 2026-07-22 | Open |

## Prevention

We will prevent similar incidents by treating routing configuration as production code: validating it against real infrastructure state, rolling it out progressively, monitoring it with config-version-level metrics, and making hub reconnect behavior less synchronized during regional failures.

The most important immediate fix is semantic config validation. A routing config that can assign customers to an empty broker pool should never reach production.