# Postmortem: Online Ordering Outage During Summer Blend Release

**Date:** 2026-07-09  
**Duration:** 1 hour 42 minutes  
**Status:** Resolved  
**Services affected:** Shopify storefront, inventory sync worker, payment checkout

## Summary

During the launch of our limited Summer Peach Oolong batch, customers were unable to complete checkout for several tea products. Product pages loaded normally, but checkout failed with inventory validation errors after customers attempted payment.

The outage was caused by a race condition between our inventory sync worker and Shopify’s stock reservation flow. A new batching change caused the worker to repeatedly overwrite reserved inventory counts with stale values from our roasting and packing system.

## Timeline

All times are Eastern Time.

- **09:00** - Summer Peach Oolong release announced by email and social channels.
- **09:07** - First customer reports checkout failure via Instagram DM.
- **09:11** - Support confirms multiple failed checkout attempts for Summer Peach Oolong and Jasmine Green.
- **09:16** - Engineering sees elevated Shopify checkout errors related to inventory mismatch.
- **09:22** - Inventory sync worker paused manually.
- **09:28** - Checkout success rate partially recovers, but some products remain unavailable.
- **09:39** - Engineering identifies stale inventory writes from the new batch sync job.
- **09:52** - Affected product inventory is manually reconciled against packed stock.
- **10:08** - Inventory sync worker restarted with batching disabled.
- **10:31** - Checkout errors return to normal baseline.
- **10:42** - Incident resolved.

## Customer Impact

- Customers could browse the storefront but could not reliably complete checkout.
- 184 checkout attempts failed.
- 73 customers were affected.
- 19 customers were charged successfully after retrying.
- No duplicate charges were created.
- The Summer Peach Oolong batch sold out later than expected due to failed checkout attempts.

Support emailed affected customers with an apology and a private reorder link. Customers who missed the release due to the outage were offered early access to the next roast.

## Root Cause

The root cause was a bug in the inventory sync worker deployed the previous afternoon.

The worker was changed to batch inventory updates every five minutes instead of processing each packing event individually. The new implementation read inventory from our internal packing system, grouped updates by SKU, and wrote the resulting counts to Shopify.

However, the worker did not account for Shopify’s temporary checkout reservations. During high traffic, Shopify reduced available inventory while customers were in checkout. The worker then overwrote those reserved counts with stale internal counts, causing Shopify to reject checkout attempts when payment completed.

This issue was not caught in testing because our staging environment does not simulate concurrent checkout reservations.

## What Went Well

- Support escalated customer reports quickly.
- Pausing the inventory sync worker reduced the error rate within minutes.
- Manual reconciliation was possible because packing counts were accurate.
- No duplicate payments or oversold orders occurred.

## What Went Wrong

- The inventory worker could overwrite Shopify inventory without checking the current remote value.
- The release checklist did not include monitoring checkout success rate.
- Staging tests did not cover high-demand product launches.
- The deployment happened less than 24 hours before a scheduled limited-batch release.

## Action Items

| Action | Owner | Due Date |
| --- | --- | --- |
| Change inventory sync to use delta-based adjustments instead of absolute overwrites | Engineering | 2026-07-16 |
| Add checkout reservation simulation to staging tests | Engineering | 2026-07-23 |
| Add checkout success-rate alerting during product launches | Engineering | 2026-07-15 |
| Freeze storefront and inventory deployments 48 hours before limited releases | Operations | 2026-07-12 |
| Add incident escalation path for social media customer reports | Support | 2026-07-18 |
| Create a launch dashboard showing checkout failures, payment success, and inventory drift | Engineering | 2026-07-26 |