# Postmortem: Home-Automation Hub Cloud Connectivity Outage **Incident date:** July 24, 2026 **Duration:** 2 hours 47 minutes **Severity:** SEV-1 **Status:** Resolved ## Summary On July 24, a configuration deployment caused the cloud gateway to reject authentication tokens issued to home-automation hubs running firmware versions earlier than 5.18. Affected hubs repeatedly attempted to reconnect, increasing load on the authentication service and delaying recovery. During the incident, customers could not reliably control devices through the mobile application, execute cloud-dependent automations, or use third-party voice assistants. Local controls and automations continued to work for most hubs. We restored service by rolling back the configuration, temporarily increasing authentication capacity, and staggering hub reconnections. ## Customer Impact Approximately 31% of active hubs lost cloud connectivity for at least five minutes, and 18% remained disconnected for more than one hour. Affected customers experienced: - Devices appearing offline in the mobile application - Failed remote commands - Delayed or missed cloud-dependent automations - Unavailable voice-assistant integrations - Delayed push notifications for alarms, locks, and sensors - Repeated hub status changes between online and offline Local device controls, physical switches, and locally stored automations remained operational unless they depended on cloud APIs. No customer data was lost, and we found no evidence of unauthorized access. ## Timeline All times are in Eastern Daylight Time. - **09:12** — A configuration update enabling stricter token validation was deployed to 10% of cloud gateway instances. - **09:17** — Deployment automation reported healthy gateway error rates and continued the rollout. - **09:24** — The configuration reached 100% of gateway instances. - **09:28** — The authentication service recorded a sharp increase in token refresh requests. No alert fired because request volume remained below the static threshold. - **09:33** — Customer support received the first reports of hubs appearing offline. - **09:39** — Monitoring detected an increase in gateway connection failures and opened a SEV-2 incident. - **09:47** — Engineers identified that failures were concentrated among hubs running firmware versions earlier than 5.18. - **09:54** — The incident was escalated to SEV-1 as authentication latency increased and newer hubs began experiencing intermittent reconnections. - **10:03** — The team halted unrelated production deployments and began rolling back the gateway configuration. - **10:16** — Rollback completed. Connection success improved, but the authentication service remained overloaded by synchronized retry traffic. - **10:24** — Authentication capacity was increased by 200%. - **10:31** — A temporary gateway rule was deployed to add randomized delays to reconnect responses. - **10:48** — Authentication latency returned to normal. - **11:06** — Most affected hubs had reconnected successfully. - **11:42** — Engineers identified a subset of hubs stuck in an expired-token retry loop and invalidated their cached sessions. - **11:59** — Cloud connectivity returned to baseline levels. - **12:11** — The incident was marked resolved after monitoring confirmed stable command success and automation execution rates. ## Root Cause The outage was caused by an incompatible token-validation change in the cloud gateway. Firmware versions earlier than 5.18 generated refresh tokens with a legacy audience field. The authentication service continued to accept these tokens under an established compatibility policy. The gateway configuration deployed during the incident enabled strict audience validation before forwarding refresh requests, causing the gateway to reject otherwise valid legacy tokens. Rejected hubs interpreted the response as a transient network failure rather than an authentication failure. They retried with exponential backoff, but their retry implementation capped the delay at 30 seconds and did not include sufficient randomization. Hundreds of thousands of hubs therefore retried in synchronized waves. This retry traffic overloaded the authentication service, increasing latency and causing some requests from unaffected firmware versions to time out. The resulting reconnect attempts amplified the load and extended the incident after the original configuration had been rolled back. ## Contributing Factors - The compatibility behavior was documented in the authentication service but not in the gateway configuration repository. - Pre-deployment tests covered current and previous firmware versions, but not older supported versions. - The canary rollout evaluated aggregate error rates and did not segment results by hub firmware. - Gateway health checks validated process availability but did not test end-to-end hub authentication. - Authentication alerts used static request-volume thresholds and did not detect abnormal refresh rates per connected hub. - Older hub firmware used insufficient retry jitter. - The rollback restored token acceptance but did not immediately reduce the accumulated retry load. ## Detection and Response The incident was first detected through gateway connection-failure monitoring, 11 minutes after the initial customer report and 21 minutes after the first measurable increase in authentication traffic. Response was slowed by dashboards that showed aggregate connection success without firmware segmentation. Once engineers correlated failures with firmware versions, they identified the configuration change quickly. Recovery took longer because rolling back the change triggered a large simultaneous reconnection wave. ## Resolution We rolled back strict audience validation at the gateway and restored the legacy-token compatibility path. We then increased authentication-service capacity, introduced randomized reconnect delays at the gateway, and invalidated cached sessions for hubs stuck in the expired-token loop. ## Action Items | Action | Owner | Priority | Target Date | |---|---|---:|---| | Add supported legacy firmware versions to gateway authentication integration tests | Hub Platform | P0 | August 7, 2026 | | Segment canary metrics by firmware version and hardware generation | Reliability Engineering | P0 | August 14, 2026 | | Require end-to-end authentication checks before gateway rollout promotion | Cloud Gateway | P0 | August 14, 2026 | | Add a compatibility-contract document shared by firmware, gateway, and authentication teams | Architecture | P1 | August 21, 2026 | | Alert on token refresh rate per connected hub and authentication rejection reason | Observability | P1 | August 21, 2026 | | Implement server-directed reconnect backoff with randomized jitter | Hub Platform | P1 | September 4, 2026 | | Remove the 30-second retry cap in supported hub firmware | Firmware | P1 | September 18, 2026 | | Add automated rollback criteria based on firmware-segmented failure rates | Release Engineering | P1 | September 18, 2026 | | Load-test authentication recovery using synchronized reconnect scenarios | Performance Engineering | P2 | October 2, 2026 | | Review alert routing so customer-support spikes can escalate active incidents automatically | Incident Management | P2 | October 2, 2026 | ## Lessons Learned Aggregate availability metrics were insufficient for a fleet containing multiple supported firmware generations. Compatibility requirements must be enforced through shared tests and deployment gates rather than team-specific documentation. We also underestimated the effect of hub retry behavior on recovery. Future changes to authentication and connectivity paths will be tested under fleet-scale reconnection conditions, including synchronized retries and degraded dependencies.