# Postmortem: Roast Control System Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 18 minutes  
**Severity:** SEV-2  
**Status:** Resolved

## Summary

On July 24, the control system for our two production tea roasters stopped accepting new roast profiles and intermittently displayed stale temperature readings. Operators halted production because they could not safely verify drum temperature or airflow.

The outage delayed 11 batches and caused three in-progress batches to be discarded. No unsafe product was shipped, and no employees were injured.

## Customer Impact

- 34 customer orders shipped one business day late.
- Two wholesale orders missed their scheduled carrier pickup.
- Three batches, totaling 18 kg of tea, were discarded.
- The tasting room and online storefront remained available.
- No customer data, payment data, or production records were lost.

Affected customers received delay notifications before the end of the business day. Shipping charges were refunded for six expedited orders.

## Timeline

All times are in Eastern Time.

| Time | Event |
|---|---|
| 08:42 | An operator reports that Roaster 2 is showing a temperature of 187°C despite the burner being off. |
| 08:45 | The shift lead pauses Roaster 2 and moves the next scheduled batch to Roaster 1. |
| 08:51 | Roaster 1 begins showing delayed sensor updates. Operators activate the emergency cooling procedure and halt both machines. |
| 08:57 | The on-call engineer confirms that the roast-control dashboard cannot reliably communicate with either roaster gateway. |
| 09:08 | Network connectivity, power, and physical sensor wiring are checked. No hardware fault is found. |
| 09:21 | Application logs show repeated database lock errors from the profile synchronization service. |
| 09:34 | The team restarts the synchronization service. Sensor updates briefly recover, then degrade again. |
| 09:47 | A recently deployed retry change is identified as the likely trigger. |
| 10:02 | The service is disabled, restoring stable live sensor readings. New roast profiles remain unavailable. |
| 10:18 | Operators verify both roasters against independent thermocouple readings. |
| 10:31 | The retry change is rolled back and the control database is restored from its pre-deployment snapshot. |
| 10:48 | Profile loading and live telemetry pass validation on both roasters. |
| 11:00 | A supervised test batch completes successfully. |
| 11:09 | Production resumes at reduced capacity. |
| 11:30 | Full production resumes, and the incident is closed. |

## Root Cause

A software update deployed the previous evening changed the profile synchronization service from a fixed retry interval to immediate retries when the local control database was busy.

During the morning profile import, one malformed supplier profile caused a transaction to fail after acquiring a write lock. The service then retried continuously without releasing the transaction correctly. Because roast profiles and recent telemetry share the same embedded database, the retry loop repeatedly blocked telemetry writes and dashboard reads.

The roaster safety controllers continued operating independently, but the dashboard displayed delayed values and could not reliably load profiles. Operators correctly treated the loss of trustworthy telemetry as unsafe and stopped production.

## Contributing Factors

- Profile synchronization and live telemetry shared a single database.
- The retry implementation had no exponential backoff or maximum attempt count.
- Integration tests did not include malformed profiles or prolonged database contention.
- Monitoring checked whether the dashboard process was running, not whether telemetry was current.
- The deployment occurred after the final roast of the day, so the failure mode did not appear until the next morning's import.
- The runbook did not document how to disable profile synchronization independently.

## Detection

The incident was detected by an operator who noticed that the displayed temperature did not match the physical state of the roaster. Automated monitoring did not alert because the dashboard and gateway processes remained healthy from the operating system's perspective.

This increased time to diagnosis and made operators the primary detection mechanism for stale telemetry.

## Resolution and Recovery

The team disabled profile synchronization to stop database lock contention, then rolled back the retry change. The local control database was restored from a snapshot taken before deployment, and the malformed supplier profile was quarantined.

Before production resumed, operators compared dashboard readings with independent thermocouples, verified airflow and burner controls, and completed a supervised test batch.

## What Went Well

- Operators stopped production promptly and followed the emergency cooling procedure.
- Independent roaster safety controls remained functional.
- The pre-deployment database snapshot allowed a quick recovery.
- Batch traceability records made it possible to identify and discard all potentially affected tea.
- Customer support notified impacted customers before carrier cutoff.

## What Went Poorly

- Monitoring did not detect stale telemetry.
- The rollback procedure required manual database restoration.
- Profile synchronization could interfere with safety-relevant operational visibility.
- The malformed profile was accepted without schema validation.
- Ownership during the first 20 minutes was unclear because the incident-response rotation was undocumented.

## Action Items

| Action | Owner | Priority | Due Date |
|---|---|---:|---|
| Add telemetry freshness alerts for each roaster gateway and dashboard | Engineering | P0 | August 3, 2026 |
| Add bounded exponential backoff and automatic circuit breaking to synchronization retries | Engineering | P0 | August 5, 2026 |
| Validate supplier profiles against a strict schema before import | Engineering | P0 | August 5, 2026 |
| Move profile synchronization and telemetry to separate databases | Engineering | P1 | August 21, 2026 |
| Add integration tests for malformed profiles, lock contention, and service restarts | Engineering | P1 | August 12, 2026 |
| Document independent shutdown and recovery procedures for synchronization | Operations | P1 | August 4, 2026 |
| Add a dashboard banner when telemetry is more than five seconds old | Engineering | P1 | August 12, 2026 |
| Formalize the incident commander and on-call rotation | Operations | P1 | August 7, 2026 |
| Run a quarterly control-system outage drill with production staff | Operations | P2 | September 30, 2026 |
| Review whether all operator-facing telemetry requires an independent fallback display | Safety Lead | P2 | August 28, 2026 |

## Lessons Learned

A process can remain technically online while being operationally unusable. For roasting, stale telemetry must be treated as unavailable telemetry, and monitoring needs to reflect that distinction.

Non-critical features such as profile synchronization must not be able to degrade operators' view of live equipment state. Future control-system changes will be evaluated not only for direct safety effects, but also for their ability to interfere with operational visibility.