# Postmortem: Roaster Control System Outage

**Incident date:** July 24, 2026  
**Duration:** 2 hours 47 minutes  
**Severity:** SEV-2  
**Systems affected:** Roaster control, production scheduling, batch traceability  
**Status:** Resolved

## Summary

On July 24, our production team lost control and monitoring access to both small-batch tea roasters. The roasters entered their designed fail-safe state, stopping active heating and preventing new batches from starting.

The outage was caused by an expired TLS certificate on the local control gateway. An incomplete certificate-renewal migration left the gateway outside automated renewal coverage. A recent controller update also removed the production team’s documented manual bypass procedure.

No unsafe temperatures or equipment damage occurred. Two active batches were discarded because their roasting profiles were interrupted, and eleven scheduled batches were delayed.

## Customer Impact

- 186 customer orders shipped one business day late.
- 27 wholesale orders missed their planned carrier pickup.
- Two limited-release batches, totaling 38 kg of tea, were discarded.
- The tasting room temporarily removed six roasted teas from its menu.
- No customers received unsafe or incorrectly labeled products.
- No customer or payment data was affected.

Customer support contacted affected wholesale customers directly and emailed retail customers whose shipments were delayed. Shipping charges were refunded for 41 expedited orders.

## Timeline

All times are in Eastern Time.

| Time | Event |
|---|---|
| 06:42 | The certificate on the local roaster-control gateway expires. Existing controller sessions continue operating. |
| 07:11 | The morning operator ends the overnight monitoring session during shift handoff. |
| 07:14 | The operator cannot reconnect to the gateway. The interface reports a generic “controller unavailable” error. |
| 07:17 | Both active roasters lose their supervisory connection and enter fail-safe mode. Heating stops automatically. |
| 07:20 | The production lead pauses new batch preparation and begins the documented manual-control procedure. |
| 07:28 | The team discovers that the manual bypass option is no longer available after the previous week’s controller update. |
| 07:34 | The production lead opens an incident and contacts the on-call operations engineer. |
| 07:48 | Initial investigation focuses on the controller update and a possible network-switch failure. |
| 08:03 | Network connectivity and controller hardware checks pass. |
| 08:16 | The on-call engineer identifies failed TLS handshakes between the operator stations and the gateway. |
| 08:22 | The expired gateway certificate is confirmed as the immediate cause. |
| 08:31 | The team attempts automated renewal, but the gateway has no valid renewal configuration. |
| 08:44 | A replacement certificate is generated from the internal certificate authority. |
| 09:02 | The replacement certificate is installed and the gateway service is restarted. |
| 09:09 | Operator access is restored. Both roasters pass idle-state and safety-interlock checks. |
| 09:18 | A dry-run profile completes successfully on Roaster 1. |
| 09:27 | A verification batch completes successfully on Roaster 2. |
| 09:34 | Production resumes at reduced capacity. The incident is resolved. |
| 14:20 | Full production capacity is restored after rescheduling batches and completing quality checks. |

## Detection

The outage was detected by a production operator when the control interface failed to reconnect. We had monitoring for gateway availability and controller connectivity, but no alert for certificate expiration or repeated TLS failures.

The gateway remained reachable at the network level, so the existing uptime check continued to report it as healthy.

## Root Cause

The direct cause was an expired TLS certificate on the local gateway that brokers commands and telemetry between operator stations and the roaster controllers.

Three months earlier, certificate management had been migrated from a manually maintained spreadsheet to an automated renewal service. The roaster gateway was omitted because it runs on an isolated production network and requires a separate deployment step. The omission was not tracked as an open migration task, and ownership of the gateway certificate remained unclear.

When the certificate expired, existing sessions remained active until the morning shift disconnected. New sessions then failed TLS validation. After losing supervisory connectivity, both roasters behaved as designed and stopped heating.

## Contributing Factors

- Certificate-expiration monitoring covered public services but not production equipment.
- The gateway health check tested network reachability rather than authenticated application access.
- The control interface displayed a generic error that did not identify certificate validation as the failure.
- A controller update deployed the previous week removed the documented local bypass option.
- The update was tested for normal operation but not for loss of gateway connectivity.
- No post-update review was performed for emergency operating procedures.
- The on-call engineer did not have a current inventory of production certificates.
- Certificate installation required a gateway restart, extending recovery time.

## What Went Well

- The roasters entered fail-safe mode and stopped heating without operator intervention.
- Temperature and airflow logs remained available from the controller’s local storage.
- The production team quickly quarantined the interrupted batches.
- Quality checks prevented partially roasted tea from entering packaging.
- Wholesale and retail teams communicated delays before scheduled shipment notifications were sent.
- Both roasters were validated with test profiles before normal production resumed.

## What Went Poorly

- We had no advance warning that the certificate was approaching expiration.
- The initial error message led responders toward network and hardware troubleshooting.
- The documented recovery procedure referenced a bypass option that no longer existed.
- Certificate ownership and renewal responsibility were not assigned.
- Recovery depended on one engineer familiar with the internal certificate authority.
- Production scheduling did not include sufficient buffer for a multi-hour equipment interruption.

## Resolution and Recovery

The operations engineer issued and installed a new gateway certificate, restarted the gateway service, and verified authenticated access from both operator stations.

Before resuming production, the team:

1. Confirmed all safety interlocks were operational.
2. Compared local controller logs with the production records.
3. Ran a dry profile without product on Roaster 1.
4. Ran a small verification batch on Roaster 2.
5. Performed sensory and moisture checks on the verification batch.
6. Rescheduled delayed batches based on order priority and tea availability.

The two interrupted batches were discarded because their roasting profiles could not be resumed within approved quality limits.

## Action Items

| Action | Owner | Priority | Due date |
|---|---|---:|---|
| Add the roaster gateway certificate to automated renewal and deployment | Operations | P0 | August 3, 2026 |
| Alert at 45, 21, 7, and 1 day before certificate expiration | Operations | P0 | August 5, 2026 |
| Replace the reachability check with an authenticated gateway health check | Engineering | P0 | August 7, 2026 |
| Assign an owner and backup owner to every production certificate | Engineering Manager | P1 | August 7, 2026 |
| Restore or replace the emergency local-control procedure | Controls Vendor | P1 | August 14, 2026 |
| Update runbooks to match the current controller interface | Production Engineering | P1 | August 14, 2026 |
| Add loss-of-gateway testing to controller update acceptance tests | Quality Assurance | P1 | August 21, 2026 |
| Improve the control interface to surface TLS and certificate errors | Controls Vendor | P2 | September 4, 2026 |
| Store certificate inventory and renewal status in the operations dashboard | Operations | P2 | September 11, 2026 |
| Train production leads to collect gateway diagnostics safely | Production Engineering | P2 | September 18, 2026 |
| Add four hours of recovery buffer to time-sensitive release schedules | Production Planning | P2 | August 14, 2026 |

## Lessons Learned

Industrial equipment can be mechanically healthy while still being unavailable because of ordinary software dependencies. Certificates, gateway services, and operator interfaces must be treated as production-critical components with the same monitoring, ownership, and recovery planning as the roasters themselves.

Fail-safe behavior prevented a safety incident, but safety alone was not sufficient for operational resilience. Future controller changes will be evaluated against both normal production workflows and degraded-mode recovery procedures.