# Postmortem: Roast Control System Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-2  
**Status:** Resolved  
**Authors:** Roastery Operations and Engineering

## Summary

On July 24, the control system for our two small-batch tea roasters stopped accepting new roast profiles and intermittently disconnected from active equipment. Production was paused to prevent inconsistent roasting and potential damage to tea or machinery.

The outage delayed 18 production batches and caused three wholesale orders to miss their scheduled carrier pickup. No unsafe equipment conditions occurred, and no customer or employee data was affected.

Service was restored after the control gateway was rolled back to the previous software release and its corrupted local configuration was rebuilt.

## Customer Impact

- Online ordering and payment remained available.
- Three wholesale orders shipped one business day late.
- Twelve subscription orders were delayed by less than 24 hours but still met their delivery estimates.
- Two limited-release teas were temporarily marked unavailable because inventory could not be replenished.
- Approximately 14 kg of tea required quality review. Of that amount, 3.5 kg was rejected because its roast profile was interrupted.
- No customers received tea that failed our quality standards.

Customer support contacted the three affected wholesale customers directly and waived their shipping charges.

## Timeline

All times are in Eastern Time.

| Time | Event |
|---|---|
| 06:42 | A new gateway release was deployed automatically before production began. |
| 07:05 | The morning operator started the first roaster and successfully loaded a roast profile. |
| 07:18 | The control dashboard displayed intermittent “roaster unavailable” warnings. |
| 07:22 | The first active roast stopped receiving profile updates. The roaster continued under its onboard temperature limits. |
| 07:24 | The operator performed a manual cooldown and paused both production lines. |
| 07:31 | Operations notified the on-call engineer. |
| 07:43 | Engineering confirmed repeated gateway restarts caused by configuration parsing failures. |
| 08:02 | The gateway was restarted, briefly restoring connectivity. |
| 08:11 | The failure returned when the second roaster requested its saved profile. |
| 08:25 | Engineering identified malformed profile metadata in the gateway’s local cache. |
| 08:41 | The team attempted to rebuild the cache from the central profile service. The rebuild failed because the new release recreated the invalid metadata. |
| 09:03 | The incident was escalated to SEV-2, and wholesale fulfillment was notified of a likely shipping delay. |
| 09:17 | Engineering decided to roll back the gateway release. |
| 09:32 | The previous version was installed, and the local configuration cache was cleared and rebuilt. |
| 09:47 | Both roasters passed connectivity and emergency-stop checks. |
| 10:01 | A test batch completed successfully. |
| 10:09 | Production resumed at reduced capacity. |
| 10:11 | The incident was declared resolved. |
| 13:40 | Normal production capacity was restored after additional batch verification. |

## Root Cause

The outage was caused by a backward-incompatible change to roast-profile metadata in the gateway release deployed that morning.

The new release converted an optional `cooldown_duration` field from an integer to a structured value containing a duration and unit. Older profiles without the field were assigned a default value, but the migration wrote that default using the previous integer format.

When a roaster requested one of these migrated profiles, the gateway failed to parse the cached configuration and restarted. Because the same invalid value was recreated during cache synchronization, restarting the service or rebuilding the cache without rolling back did not resolve the problem.

The roasters’ onboard safety controllers continued operating independently. They enforced temperature limits and allowed operators to initiate manual cooldowns, preventing an unsafe condition.

## Contributing Factors

- The migration test suite covered profiles with explicit cooldown settings but not older profiles where the field was absent.
- The release was deployed automatically shortly before the production shift, leaving little observation time before equipment use.
- The gateway treated a single invalid cached profile as a fatal startup error instead of rejecting only that profile.
- Restart alerts were configured for five failures within ten minutes. The gateway initially restarted fewer times than this threshold, delaying the engineering alert.
- The rollback procedure required manual cache cleanup, which was documented but not included in the deployment tooling.
- The control dashboard reported generic connectivity failures and did not display the configuration parsing error.

## Detection

The incident was first detected by the roastery operator through dashboard warnings and interrupted profile updates. Automated monitoring did not page the on-call engineer before operations reported the issue.

This increased detection time by approximately nine minutes and made the initial symptoms appear to be a network problem rather than a configuration failure.

## Resolution and Recovery

Engineering restored service by:

1. Pausing production and placing both roasters in a safe cooldown state.
2. Rolling the gateway software back to the previous release.
3. Clearing the incompatible local configuration cache.
4. Rebuilding the cache from the central profile service using the previous schema.
5. Running connectivity, safety-interlock, and profile-loading checks.
6. Completing a test roast before resuming production.

Operations prioritized delayed wholesale batches after recovery and added a second shift to complete the remaining subscription orders.

## What Went Well

- Operators followed the equipment-fault procedure and stopped production immediately.
- Independent roaster safety controls functioned as designed.
- Lot tracking allowed the affected tea to be isolated quickly.
- Engineering reproduced the failure using a copied profile without modifying production data.
- Operations, fulfillment, and customer support coordinated shipping communications before the carrier cutoff.

## What Went Poorly

- Automated monitoring did not identify the failure quickly enough.
- The deployment occurred too close to the start of production.
- The dashboard obscured the underlying parsing error.
- The rollback process depended on undocumented knowledge about cache cleanup.
- Compatibility testing did not include a representative sample of historical roast profiles.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add migration tests for missing, null, legacy, and malformed profile fields. | Software Engineering | P0 | August 3, 2026 |
| Change the gateway to quarantine invalid profiles instead of restarting. | Software Engineering | P0 | August 10, 2026 |
| Validate all stored profiles against the new schema before deployment. | Platform Engineering | P0 | August 5, 2026 |
| Add an automated rollback step that restores the compatible cache format. | Platform Engineering | P1 | August 14, 2026 |
| Block production-control deployments within two hours of a roasting shift unless manually approved. | Engineering Manager | P1 | August 7, 2026 |
| Page on the first repeated configuration parsing failure. | Platform Engineering | P1 | August 3, 2026 |
| Display gateway error categories and affected profile IDs in the operations dashboard. | Software Engineering | P1 | August 17, 2026 |
| Add historical production profiles to the pre-release compatibility test set. | Quality Engineering | P1 | August 7, 2026 |
| Run a quarterly roaster-control rollback drill with operators and engineering. | Roastery Operations | P2 | September 30, 2026 |
| Update the incident runbook with cache recovery and verification procedures. | Site Reliability Engineering | P2 | August 5, 2026 |

## Lessons Learned

Our equipment remained safe because critical protections run on the roasters rather than the networked control gateway. However, production availability still depended on the gateway accepting every cached profile.

Future releases must assume that historical configuration data may be incomplete or malformed. Invalid profile data should reduce functionality for the affected batch, not take the entire roasting system offline.