# Postmortem: Home Automation Hub Cloud Sync Outage

**Date:** 2026-07-08  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-2  
**Status:** Resolved

## Summary

On July 8, 2026, a backend deployment caused a subset of home-automation hubs to lose cloud connectivity. Local automations continued to run, but users could not reliably control devices through the mobile app, receive push notifications, or use voice-assistant integrations.

The outage was caused by a schema migration that changed hub session validation behavior without maintaining compatibility for hubs running older firmware. The issue was resolved by rolling back the authentication service deployment and restoring the previous validation path.

## Customer Impact

Approximately 38% of active hubs were affected.

Affected customers experienced:

- Mobile app showing hubs as offline
- Delayed or failed device commands from outside the home
- Voice-assistant commands failing for connected devices
- Missing security, water leak, and motion push notifications
- Failed remote access to live camera thumbnails

Not affected:

- Local device control on the same network
- Scheduled automations already stored on the hub
- Zigbee, Z-Wave, Thread, and Matter device communication inside the home

## Timeline

All times are in UTC.

- **14:02** - Authentication service deployment started.
- **14:09** - First increase in hub reconnect attempts detected.
- **14:13** - Mobile API error rate exceeded alert threshold.
- **14:18** - Support began receiving reports of hubs appearing offline.
- **14:24** - Incident declared as SEV-2.
- **14:31** - Engineering identified elevated `401` responses from hub session refresh requests.
- **14:45** - Deployment was paused while logs and recent changes were reviewed.
- **15:02** - Team isolated failures to hubs running firmware versions prior to `5.12.0`.
- **15:18** - Rollback of authentication service began.
- **15:31** - Hub reconnect success rate began recovering.
- **15:54** - Mobile API error rate returned to normal levels.
- **16:19** - Remaining delayed push notification queues drained.
- **16:26** - Incident marked resolved.

## Root Cause

The authentication service deployment introduced stricter validation for hub session refresh tokens. The new validation logic assumed all hubs included a `firmware_capabilities` claim during session renewal.

Hubs running firmware `5.11.x` and earlier did not include this claim. Instead of treating the missing claim as an older-client compatibility case, the service rejected the refresh request as invalid.

This caused affected hubs to repeatedly attempt session renewal and fail. Once their existing session expired, they appeared offline to cloud services even though the physical hubs remained operational on the local network.

## Contributing Factors

- Compatibility tests covered mobile clients but did not include older hub firmware versions.
- The rollout was performed globally instead of by region or firmware cohort.
- The alert for hub reconnect failures triggered later than the mobile API error-rate alert.
- The migration plan documented database compatibility but did not explicitly cover token payload compatibility.

## Resolution

The authentication service was rolled back to the previous version. Once restored, affected hubs successfully refreshed their sessions and reconnected to cloud services.

No hub firmware update or customer action was required.

## What Went Well

- Local automations continued running during the outage.
- Rollback was completed quickly after the root cause was identified.
- Support and engineering coordination helped correlate customer reports with backend telemetry.
- No customer configuration or device-pairing data was lost.

## What Went Poorly

- The deployment reached all production regions before compatibility risk was detected.
- Existing tests did not represent enough real-world hub firmware versions.
- Customer-facing status-page updates lagged behind internal incident updates.
- Push notification delays continued after hub connectivity recovered because queues had backed up.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add authentication compatibility tests for all supported hub firmware versions | Platform | P0 | 2026-07-15 |
| Require staged rollout by region and firmware cohort for auth changes | Release Engineering | P0 | 2026-07-18 |
| Add alerting on hub session refresh failure rate | Observability | P1 | 2026-07-19 |
| Update deployment checklist to include token and protocol compatibility review | Platform | P1 | 2026-07-22 |
| Improve status-page automation for SEV-1 and SEV-2 incidents | Support Engineering | P2 | 2026-07-31 |
| Add replay-safe push notification queue draining during hub reconnect events | Cloud Services | P2 | 2026-08-05 |

## Prevention

Future authentication and protocol changes will be tested against the full supported firmware matrix before production rollout. Changes affecting hub connectivity will also use staged deployment with automatic rollback if reconnect failures, session refresh failures, or hub offline rates exceed defined thresholds.