# Postmortem: Home Automation Hub Command Processing Outage **Incident date:** July 24, 2026 **Duration:** 1 hour 47 minutes **Severity:** SEV-1 **Affected systems:** Cloud-connected home automation hubs, mobile applications, voice-assistant integrations **Status:** Resolved ## Summary On July 24, 2026, a production deployment caused home automation hubs to repeatedly reconnect to the cloud command service. The resulting connection surge exhausted the service’s database connection pool and delayed or prevented commands from reaching customer devices. During the incident, customers experienced failed or delayed device control through the mobile application, automations, and voice assistants. Local controls, including physical switches and automations configured to run entirely on the hub, continued to operate. We restored service by rolling back the deployment, temporarily increasing connection capacity, and gradually reconnecting hubs to avoid a second traffic spike. ## Customer Impact Between 18:12 and 19:59 UTC: - Approximately 38% of online hubs experienced at least one cloud disconnection. - Roughly 21% of cloud-issued device commands failed or timed out. - Successful commands had a median delay of 8 seconds and a 95th-percentile delay of 74 seconds. - Scheduled automations requiring cloud evaluation were delayed by up to 26 minutes. - Voice-assistant integrations frequently reported that devices were unavailable. - Device status shown in the mobile application was sometimes stale. Safety-critical local features, including smoke and carbon-monoxide alarms, were not affected. Door locks and garage-door controllers remained operable through physical controls and local hub automations, but remote commands to those devices were subject to the same failures and delays. ## Detection The incident was first detected by an alert on elevated hub reconnect rates at 18:15 UTC. A separate alert for command-processing latency fired three minutes later. Customer support began receiving reports at 18:16 UTC. Our public status page was updated at 18:27 UTC. The deployment pipeline’s automated checks did not detect the issue because pre-production environments did not contain enough concurrently connected hubs to reproduce the reconnect amplification. ## Timeline All times are in UTC on July 24, 2026. - **17:54** — Version `command-gateway-2026.07.24.3` begins rolling out to 10% of production instances. - **18:03** — Deployment reaches 100%. Error rates remain within normal thresholds. - **18:09** — A routine certificate-cache refresh occurs across gateway instances. - **18:12** — Hubs begin reconnecting at approximately six times the normal rate. - **18:15** — Reconnect-rate alert fires. The on-call engineer begins investigation. - **18:16** — Customer support reports an increase in “hub offline” and command-timeout contacts. - **18:18** — Command latency exceeds the SEV-1 threshold. - **18:21** — Incident commander declares a SEV-1 and assembles the response team. - **18:27** — Public status page updated to acknowledge degraded cloud connectivity. - **18:31** — Engineers identify database connection-pool exhaustion in the command gateway. - **18:38** — Database connection limits are increased as a temporary mitigation. Command throughput improves briefly but reconnect traffic continues growing. - **18:44** — The team correlates the reconnect surge with the gateway deployment and begins rollback. - **18:53** — Rollback completes on all gateway instances. - **18:57** — Reconnect rates begin falling, but synchronized retries from hubs continue to overload several instances. - **19:05** — A temporary rate limit is applied to hub authentication requests. - **19:13** — Hubs are admitted in controlled batches. Command success rates begin recovering. - **19:32** — Command-processing latency returns below the SEV-1 threshold. - **19:47** — More than 99% of previously connected hubs are online. - **19:59** — Metrics remain stable for 12 minutes. Incident marked resolved. - **20:18** — Status page updated to indicate full recovery. - **22:10** — Deferred scheduled automations finish processing. ## Root Cause The outage was caused by a defect introduced in `command-gateway-2026.07.24.3`. The command gateway maintains long-lived connections with customer hubs. When a gateway refreshes its certificate cache, each active connection performs a lightweight authentication check. The new release changed that check to load household authorization data directly from the primary database instead of using the existing in-memory cache. The same release also incorrectly interpreted a slow authentication response as an expired hub session. Affected hubs closed their connections and retried immediately. Because retries had insufficient random delay, large groups of hubs reconnected at nearly the same time. This created a positive feedback loop: 1. Certificate refreshes triggered database-backed authentication checks. 2. The database connection pool became saturated. 3. Authentication responses slowed down. 4. Gateways treated slow responses as expired sessions and disconnected hubs. 5. Hubs reconnected simultaneously, generating additional authentication queries. 6. Increased query volume further saturated the connection pool. The database itself remained healthy, but the command gateway could not acquire database connections quickly enough to authenticate hubs or process customer commands. ## Contributing Factors - The authentication path used a fixed timeout that was too aggressive under elevated load. - Hub reconnect logic used a narrow retry interval with inadequate jitter. - Load tests modeled command traffic but not millions of concurrent connection refreshes. - The rollout proceeded from 10% to 100% before a certificate-cache refresh exercised the affected code path. - The deployment health check focused on request error rates and did not evaluate reconnect frequency or database-pool wait time. - The command gateway and command processor shared the same database connection pool, allowing authentication traffic to starve command execution. - The operational runbook recommended increasing database capacity before applying authentication rate limits, which provided only temporary relief. ## Resolution and Recovery We rolled back the command gateway to the previous release, restoring cached authorization checks and the prior session-timeout behavior. Because many hubs continued retrying in sync after the rollback, we also rate-limited authentication requests and gradually increased the admission rate. This allowed existing commands to drain while hubs re-established stable connections. Delayed scheduled automations were retained in the queue and processed after interactive command traffic recovered. Commands that had already timed out were not automatically retried to avoid unexpected device actions after customer context had changed. ## What Went Well - Reconnect-rate and command-latency alerts fired within six minutes of customer impact. - The incident response team identified the responsible deployment within 32 minutes. - Local hub functionality continued operating throughout the outage. - Command queues preserved scheduled automations without data loss. - Support, engineering, and communications teams maintained a shared incident log and consistent customer messaging. ## What Went Poorly - The deployment appeared healthy until a periodic certificate-cache refresh exercised the defective path. - The canary stage was too short to include that refresh cycle. - Authentication and command execution competed for the same constrained resource. - Status-page updates lagged the first customer reports by 11 minutes. - The initial capacity increase delayed application-level mitigation. - We lacked a safe, automated mechanism for gradually reconnecting hubs after a large-scale disconnect. ## Corrective and Preventive Actions | Action | Owner | Priority | Due date | Status | |---|---|---:|---|---| | Restore cached authorization lookups and add tests preventing database access in connection-refresh paths | Hub Platform | P0 | August 4, 2026 | In progress | | Separate authentication and command-processing database connection pools | Cloud Infrastructure | P0 | August 11, 2026 | In progress | | Add exponential backoff with full jitter to hub reconnection logic | Hub Firmware | P0 | August 18, 2026 | Planned | | Extend canary observation to cover certificate refresh and other periodic jobs | Release Engineering | P1 | August 7, 2026 | In progress | | Add deployment gates for reconnect rate, connection-pool wait time, and authentication latency | Reliability Engineering | P1 | August 14, 2026 | Planned | | Build a load-test scenario for concurrent certificate refreshes and mass reconnects | Performance Engineering | P1 | August 21, 2026 | Planned | | Implement automatic authentication admission control during reconnect surges | Hub Platform | P1 | August 28, 2026 | Planned | | Change slow authentication responses to fail without invalidating otherwise valid sessions | Identity Platform | P1 | August 11, 2026 | In progress | | Add a one-command rollback and staged hub readmission procedure to the incident runbook | Reliability Engineering | P1 | August 5, 2026 | Completed | | Review status-page escalation timing and automate initial degraded-service notices | Customer Operations | P2 | August 14, 2026 | Planned | | Add quarterly resilience testing for loss and restoration of all hub connections | Reliability Engineering | P2 | September 30, 2026 | Planned | ## Lessons Learned Long-lived device connections behave differently from ordinary web traffic. A small change in session handling can affect millions of clients simultaneously, and client retry behavior can amplify a partial slowdown into a full service outage. Future releases affecting authentication, connection lifecycle, certificates, or retry behavior will require extended canary periods and dedicated reconnect testing. We are also isolating command execution from connection-management workloads so that a hub connectivity event cannot consume the resources required to process customer actions.