# Postmortem: Automation Execution Delays and Device Control Failures

**Incident date:** July 24, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-1  
**Status:** Resolved

## Summary

On July 24, the home-automation hub experienced widespread delays and failures when executing automations and remote device commands. Local manual controls continued to work for most devices, but cloud-dependent features—including mobile app commands, voice-assistant requests, notifications, and remotely triggered routines—were degraded.

The incident began after a routine deployment changed how hub connection state was written to the session database. The new code produced substantially more writes than expected. Under peak evening traffic, the database reached its connection limit, causing requests to queue and time out. Automatic retries amplified the load and delayed recovery.

We restored service by rolling back the deployment, disabling nonessential connection-state updates, and gradually draining queued automation jobs.

## Customer Impact

Approximately 31% of online hubs were affected during the incident.

Affected customers experienced one or more of the following:

- Remote commands from the mobile app failed or took up to 12 minutes to execute.
- Cloud-triggered automations ran late, ran after customers had retried them, or did not run.
- Voice-assistant integrations returned timeout or unavailable responses.
- Push notifications for alarms, locks, water sensors, and door sensors were delayed.
- Hub status appeared offline or stale in the mobile app.
- Some customers received duplicate notifications after service recovered.

Safety-critical monitoring performed locally by the hub continued to operate. Locally configured sirens, leak-shutoff rules, and device-to-device automations were not affected. However, delayed cloud notifications reduced customers’ ability to respond remotely.

No customer data was lost or exposed. We identified 4,812 automation jobs that exceeded their execution window and were discarded rather than replayed to avoid unsafe late actions, such as unlocking doors or opening garage doors.

## Timeline

All times are in Eastern Time.

- **18:42** — Version `hub-session-api-2026.07.24.3` was deployed to 10% of production.
- **18:49** — The deployment reached 100% after initial error-rate and latency checks remained within thresholds.
- **19:07** — Database write volume began increasing as North American evening traffic grew.
- **19:18** — The session database reached 85% connection utilization. No alert fired because the threshold was configured at 95%.
- **19:24** — Customer command latency exceeded the service-level objective.
- **19:27** — The first automated alert fired for elevated API timeout rates.
- **19:31** — The on-call engineer began investigating the command gateway and automation queue.
- **19:38** — Customer Support reported a rise in complaints about hubs appearing offline.
- **19:44** — Engineers identified database connection exhaustion in the hub session service.
- **19:51** — The team increased the database connection limit as an immediate mitigation. Error rates briefly improved.
- **20:03** — Retry traffic consumed the added capacity, and timeout rates rose again.
- **20:09** — The incident was escalated to SEV-1. Deployments were frozen.
- **20:17** — Engineers correlated the increase in writes with the session-state deployment.
- **20:24** — Rollback began.
- **20:36** — The rollback completed across all regions. New command success rates began recovering.
- **20:43** — Nonessential presence and connection-state writes were disabled with a feature flag.
- **20:51** — Database connection utilization returned below 60%.
- **21:02** — The automation queue began draining. Potentially unsafe expired actions were discarded.
- **21:19** — Mobile app and voice-assistant success rates returned to normal.
- **21:41** — The remaining notification backlog was processed.
- **21:44** — The incident was declared resolved.
- **22:30** — Engineers confirmed that local safety automations had continued operating and that no customer data had been lost.

## Root Cause

The deployment modified hub presence tracking to persist a `last_seen_at` timestamp whenever a hub sent a heartbeat or device-state message. Previously, these updates were coalesced in memory and written at most once every five minutes per hub.

A refactoring inadvertently bypassed that coalescing path. As a result, each message generated a synchronous database update. Active hubs typically send several messages per minute, so production write volume increased by approximately 14 times during peak traffic.

The session service shared a database connection pool with the command-routing service. Once the pool was exhausted, command requests waited for connections and eventually timed out. Clients and internal workers retried timed-out requests with nearly identical backoff intervals, creating synchronized retry bursts that further increased load.

## Contributing Factors

- Load tests modeled average hub traffic but did not include peak heartbeat and device-event rates.
- The deployment canary occurred before peak evening usage and was evaluated for only seven minutes.
- Database write volume was visible on dashboards but did not have a deployment-blocking alert.
- Presence updates and command routing shared the same database and connection pool.
- Retry policies lacked sufficient jitter and did not enforce a global retry budget.
- Increasing the connection limit treated the symptom and briefly intensified database contention.
- The rollback procedure required a full application redeployment rather than an immediate configuration change.
- Queue-age alerts were configured for five minutes, which was too slow for time-sensitive automations.

## Resolution and Recovery

We rolled back the session-service deployment and disabled nonessential presence writes. This reduced database traffic and allowed command-routing requests to obtain connections normally.

During recovery, we classified queued jobs by safety and expiration policy. Notifications and idempotent state-sync operations were replayed. Commands with potentially unsafe effects—such as unlocking doors, opening garage doors, or disabling alarms—were discarded if their intended execution window had passed.

We then drained the remaining queues gradually to prevent another traffic spike.

## What Went Well

- Local automations continued operating because hub execution does not depend on the cloud after rules are synchronized.
- Existing expiration metadata allowed us to prevent unsafe delayed commands from being replayed.
- The feature flag for presence writes provided a second mitigation after rollback.
- Engineering, Customer Support, and Security established a shared incident channel within 20 minutes of detection.
- Regional metrics made it possible to confirm that the issue was systemic rather than isolated to a network provider.

## What Went Poorly

- Monitoring detected API symptoms rather than the underlying database saturation.
- The canary window was too short and did not overlap representative traffic.
- Retry behavior was not included in capacity testing.
- The initial database connection-limit increase delayed identification of the actual cause.
- Customer-facing status updates began 38 minutes after measurable impact.
- Support documentation did not clearly explain which automations continue to run locally during a cloud outage.

## Action Items

| Action | Owner | Priority | Target Date |
|---|---|---:|---|
| Restore coalesced presence updates and add regression tests for maximum write frequency per hub | Hub Platform | P0 | August 3, 2026 |
| Separate command-routing and presence workloads into independent connection pools | Cloud Infrastructure | P0 | August 7, 2026 |
| Add database write-rate and connection-wait alerts tied to deployment automation | Reliability Engineering | P0 | August 5, 2026 |
| Add exponential backoff with jitter and retry budgets to clients and internal workers | Device Connectivity | P0 | August 14, 2026 |
| Extend canary evaluation to 30 minutes and require representative peak-load validation | Release Engineering | P1 | August 10, 2026 |
| Add peak heartbeat, reconnect-storm, and retry-amplification scenarios to load testing | Performance Engineering | P1 | August 21, 2026 |
| Reduce automation queue-age alert thresholds from five minutes to 60 seconds | Automation Platform | P1 | August 4, 2026 |
| Create per-command expiration policies and audit all remotely executable safety-sensitive actions | Product Security | P1 | August 28, 2026 |
| Make nonessential session writes dynamically rate-limited through configuration | Hub Platform | P1 | August 18, 2026 |
| Update the public status process to publish an initial notice within 15 minutes of SEV-1 declaration | Customer Operations | P1 | August 7, 2026 |
| Document local-versus-cloud automation behavior for customers and Support | Technical Writing | P2 | August 21, 2026 |
| Run a recovery exercise covering queue draining and expired-command handling | Reliability Engineering | P2 | September 4, 2026 |

## Lessons Learned

Cloud degradation must not turn routine telemetry into contention with customer commands. Presence data is useful, but it is less important than timely execution of control and safety-related workflows. We are isolating these workloads, tightening deployment safeguards, and treating retry behavior as part of normal capacity planning rather than an exceptional condition.