# Postmortem: Home-Automation Hub Command Dispatch Outage

**Date:** 2026-07-09  
**Duration:** 1 hour 47 minutes  
**Severity:** SEV-2  
**Status:** Resolved

## Summary

On July 9, 2026, HomeHub users experienced delayed or failed automation commands across cloud-connected devices. Commands issued from the mobile app, voice assistants, and scheduled routines were accepted by the API but were not reliably delivered to hubs.

Local automations continued to run for most customers, but cloud-triggered actions such as remote lock control, lighting scenes, thermostat changes, and security mode updates were affected.

## Customer Impact

- Approximately 38% of active hubs experienced command delays or failures.
- Remote device control from the mobile app was unreliable.
- Voice-assistant commands intermittently timed out.
- Scheduled cloud routines were delayed by up to 45 minutes.
- Local-only automations continued running unless they depended on cloud state.
- No evidence of data loss, unauthorized access, or unsafe device state changes was found.

## Timeline

All times are UTC.

| Time | Event |
|---|---|
| 14:06 | Deployment `command-router-2026.07.09.3` completed in production. |
| 14:13 | Command queue depth began increasing in `us-east-1`. |
| 14:19 | First elevated timeout alerts fired for command dispatch latency. |
| 14:24 | Customer support began receiving reports of failed app and voice commands. |
| 14:31 | On-call engineer declared SEV-2 and started incident bridge. |
| 14:38 | Rollback initiated for `command-router`. |
| 14:44 | Rollback completed, but queue depth continued increasing. |
| 14:51 | Investigation found dispatch workers repeatedly reconnecting to MQTT broker shards. |
| 15:07 | Feature flag `hub_presence_fast_path` disabled globally. |
| 15:16 | Worker reconnect rate dropped to baseline. |
| 15:29 | Queue drain rate exceeded enqueue rate. |
| 15:53 | Command latency returned below 2 seconds for 95% of requests. |
| 16:00 | Incident marked resolved. |
| 17:30 | Backlog audit completed; no stuck commands remained. |

## Root Cause

The outage was caused by a regression in the new hub presence fast path introduced in `command-router-2026.07.09.3`.

The new code cached hub-to-broker shard mappings to reduce lookup latency. Under normal hub reconnect behavior, mappings were expected to expire within 30 seconds. However, the cache key did not include the broker shard generation ID. When a broker shard restarted or rebalanced, command-router workers continued using stale shard mappings.

This caused dispatch attempts to fail with reconnectable MQTT errors. The retry logic treated these errors as transient and immediately retried, which increased worker saturation and delayed healthy commands behind retry traffic.

## Contributing Factors

- The staging environment did not simulate broker shard rebalancing during command-router load tests.
- Retry backoff for MQTT dispatch errors was too aggressive.
- Queue depth alerts fired quickly, but there was no alert specifically for stale hub presence mappings.
- The rollback did not immediately resolve the incident because poisoned retry traffic remained in the queue.

## What Went Well

- API availability remained stable.
- Local hub automations continued working for most customers.
- The incident was detected within 13 minutes of the deployment.
- Disabling the feature flag mitigated the issue without a full service restart.

## What Went Poorly

- The rollback path was slower than expected because queued retries continued consuming worker capacity.
- Customer-facing status updates lagged behind support ticket volume.
- The deployment health check focused on API success rate, not end-to-end command delivery.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Include broker shard generation ID in hub presence cache keys. | Platform | P0 | 2026-07-15 |
| Add integration tests for broker shard restart and rebalance scenarios. | QA Automation | P0 | 2026-07-18 |
| Add exponential backoff and jitter for MQTT dispatch retries. | Platform | P0 | 2026-07-16 |
| Add alert for stale hub presence mapping error rate. | SRE | P1 | 2026-07-20 |
| Update deployment health checks to validate end-to-end command delivery. | Release Engineering | P1 | 2026-07-22 |
| Add queue quarantine behavior for repeatedly failing command batches. | Platform | P1 | 2026-07-25 |
| Improve customer status page automation for SEV-2 incidents. | Support Engineering | P2 | 2026-07-29 |

## Preventive Measures

Future command-router releases that modify hub routing, presence, or retry behavior must pass broker rebalance tests before production rollout. Canary deployments will also include synthetic command delivery checks against test hubs in each production region.

## Current Status

The feature flag remains disabled while the cache key fix and retry changes are completed. All delayed commands have either been delivered or expired according to normal command TTL behavior. No customer action is required.