# Postmortem: Roast Control System Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 17 minutes  
**Severity:** SEV-2  
**Status:** Resolved

## Summary

On July 24, the control system for our two production tea roasters became unavailable during the morning roasting shift. Operators could not load roast profiles, monitor batch temperatures, or record roast data. Production was paused while the team diagnosed the issue and restored the service from a known-good configuration.

The outage was caused by an expired TLS certificate on the local message broker used by the roaster controllers and monitoring application. A recent configuration change had enabled strict certificate validation on the controllers, but the broker certificate was not included in our certificate-renewal process.

No unsafe equipment behavior occurred. The roasters entered their designed fail-safe state and shut off heat automatically when controller communication was lost.

## Customer Impact

- Seven wholesale orders missed the same-day roasting cutoff.
- Three orders shipped one business day late.
- Nine scheduled batches were delayed; two were cancelled because staffing was unavailable after recovery.
- One 12 kg batch of oolong in progress at the start of the incident was discarded because its roast profile could not be completed consistently.
- The production team spent approximately six staff-hours on diagnosis, recovery, cleanup, and rescheduling.
- No customers received incorrectly roasted or untraceable product.

Affected customers were notified before the daily carrier pickup. No retail storefront or online ordering functions were affected.

## Timeline

All times are in Eastern Time.

| Time | Event |
|---|---|
| 07:42 | An operator starts the first roast of the morning on Roaster 1. |
| 07:46 | The operator dashboard stops receiving temperature and airflow updates. Roaster 1 enters its fail-safe state and disables heat. |
| 07:48 | Roaster 2 rejects a new roast profile with a connection error. Production is paused. |
| 07:53 | The shift lead restarts the dashboard workstation. The issue persists. |
| 08:02 | The on-call engineer is contacted and begins remote diagnosis. |
| 08:14 | Network connectivity to both roaster controllers is confirmed. The dashboard and controllers are found to be repeatedly failing TLS handshakes with the message broker. |
| 08:27 | Logs identify an expired broker certificate. |
| 08:39 | The team attempts certificate renewal using the standard internal script. The attempt fails because the broker host is not included in the renewal inventory. |
| 08:51 | A new certificate is generated manually and installed on the broker. |
| 09:03 | The broker restarts, but Roaster 2 continues rejecting the certificate because its trust store contains an outdated internal certificate authority. |
| 09:18 | The current certificate-authority chain is installed on both roaster controllers. |
| 09:29 | Telemetry resumes from both roasters. Test profiles are loaded without errors. |
| 09:41 | Operators complete dry-run safety checks and verify temperature, airflow, alarm, and emergency-stop signals. |
| 09:59 | Production resumes on Roaster 1. |
| 10:03 | Production resumes on Roaster 2. The incident is closed. |

## Detection

The incident was detected by an operator when the dashboard stopped updating and the active roaster disabled heat. The monitoring system generated broker connection alerts, but those alerts were sent only to an email inbox that is not monitored continuously during production hours.

Time to operator detection was approximately two minutes. Time to engage engineering was 17 minutes.

## Root Cause

The immediate cause was an expired TLS certificate on the local MQTT message broker that carries commands and telemetry between the roaster controllers and the operator dashboard.

Two weeks before the incident, strict certificate validation had been enabled on both controllers as part of a security hardening change. Previously, the controllers logged certificate errors but continued connecting. After the change, the expired certificate correctly caused them to reject the broker connection.

The broker certificate had been issued manually during the original system installation and was absent from the automated certificate-renewal inventory. Its expiration date was also not tracked by monitoring. As a result, the certificate expired without renewal or warning.

Recovery took longer because Roaster 2 had missed an earlier trust-store update and did not recognize the current internal certificate authority used to issue the replacement certificate.

## Contributing Factors

- The security hardening change was tested with valid certificates but did not include an expired-certificate scenario.
- The change review did not verify that all affected certificates were monitored and automatically renewed.
- Certificate ownership was undocumented.
- Roaster 2 had no automated configuration-compliance checks.
- Broker alerts were routed to email rather than the production paging channel.
- Operators had no documented procedure for distinguishing a dashboard failure from a controller-to-broker communication failure.
- The system had no supported offline mode for loading a previously approved roast profile.

## Resolution and Recovery

The team generated and installed a replacement broker certificate, updated the certificate-authority trust stores on both roaster controllers, and restarted the broker and controller communication services.

Before production resumed, operators performed dry-run checks on both roasters and verified that telemetry, profile loading, alarms, heat interlocks, and emergency stops behaved as expected. The partially roasted batch was quarantined and later discarded under the existing quality procedure.

## What Went Well

- Both roasters entered their fail-safe state and disabled heat as designed.
- Operators paused production rather than attempting to bypass the control system.
- Controller and broker logs made the TLS failure identifiable without physical disassembly or vendor support.
- Production and customer-support staff coordinated quickly enough to notify affected wholesale customers before pickup.
- Batch traceability records clearly identified the product requiring disposal.

## What Went Poorly

- Certificate expiration was neither monitored nor included in automated renewal.
- Alert routing did not match the operational urgency of a production-stopping failure.
- Configuration drift on Roaster 2 added 27 minutes to recovery.
- The hardening change increased failure sensitivity without a corresponding review of certificate lifecycle management.
- The recovery procedure depended on knowledge held by one engineer and had not been exercised by the operations team.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add all broker and controller certificates to the automated renewal inventory. | Engineering | P0 | August 3, 2026 |
| Alert the on-call channel at 30, 14, and 7 days before certificate expiration. | Engineering | P0 | August 3, 2026 |
| Add an end-to-end synthetic check for profile loading and telemetry flow. | Engineering | P0 | August 7, 2026 |
| Route production-stopping broker and controller alerts to the paging channel. | Engineering | P0 | August 3, 2026 |
| Automate controller trust-store deployment and configuration-drift reporting. | Engineering | P1 | August 14, 2026 |
| Document certificate ownership, renewal, replacement, and rollback procedures. | Engineering | P1 | August 10, 2026 |
| Create an operator runbook for dashboard and controller connectivity failures. | Production | P1 | August 10, 2026 |
| Add certificate-expiration and trust-store checks to security-change reviews. | Engineering | P1 | August 14, 2026 |
| Conduct a recovery drill with production operators and the secondary on-call engineer. | Operations | P2 | August 21, 2026 |
| Evaluate a read-only offline mode for running previously approved roast profiles safely. | Engineering | P2 | September 4, 2026 |

## Lessons Learned

Security controls and operational reliability must be reviewed together. Enabling strict certificate validation was the correct security decision, but it exposed an unmanaged dependency in certificate issuance, renewal, and trust distribution. Future changes to authentication, encryption, or trust settings will require explicit lifecycle checks, failure-mode testing, monitoring coverage, and a documented recovery path.