# Postmortem: Checkout Outage During Saturday Market Rush

**Date:** 2026-07-04  
**Duration:** 54 minutes  
**Severity:** SEV-1  
**Status:** Resolved  
**Services affected:** Marketplace web app, mobile checkout, vendor inventory sync

## Summary

On Saturday morning, shoppers were unable to complete purchases for pickup orders across multiple farmers-market locations. Browsing and search continued to work, but checkout requests either timed out or returned inventory errors.

The outage was caused by a database connection pool exhaustion issue triggered by a new inventory reservation workflow deployed the previous evening. During peak traffic, each checkout attempt opened multiple long-running transactions while waiting on vendor inventory locks. The API exhausted available database connections, causing checkout failures and delayed vendor inventory updates.

## Customer Impact

- 38% of checkout attempts failed during the incident window.
- Approximately 1,240 shoppers saw checkout errors or timeouts.
- 312 completed carts were abandoned and not retried.
- 47 vendors saw delayed inventory updates in their dashboards.
- Some shoppers received “item unavailable” messages for products that were still in stock.
- No confirmed orders were lost or double-charged.

## Timeline

All times are in Eastern Time.

| Time | Event |
| --- | --- |
| 08:42 | New Saturday market traffic begins ramping up. |
| 08:51 | Checkout latency exceeds 5 seconds at p95. |
| 08:56 | First customer support tickets report failed checkout. |
| 08:59 | Alert fires for elevated checkout error rate. |
| 09:03 | On-call engineer confirms API database connection pool saturation. |
| 09:08 | Incident declared SEV-1. |
| 09:12 | Recent inventory reservation deployment identified as likely trigger. |
| 09:18 | Checkout service scaled horizontally, but errors continue because database connections remain exhausted. |
| 09:25 | Feature flag disabled for the new reservation workflow. |
| 09:31 | Database connection usage begins recovering. |
| 09:38 | Checkout error rate returns near baseline. |
| 09:46 | Vendor inventory sync backlog clears. |
| 09:50 | Incident resolved. |
| 10:15 | Support sends recovery message to affected shoppers with active carts. |

## Root Cause

The direct cause was database connection pool exhaustion in the checkout API.

A new inventory reservation workflow introduced a transaction pattern that held database connections while checking product availability, reserving cart items, and updating vendor inventory rows. Under normal load this passed testing, but during Saturday market peak traffic, many carts contained items from several vendors. That caused checkout requests to acquire more row locks and hold connections longer than expected.

Once the checkout API exhausted its database connection pool, incoming checkout requests queued until they timed out. Retrying shoppers increased request volume, which made recovery slower.

## Contributing Factors

- Load tests did not model multi-vendor baskets during peak Saturday traffic.
- The feature shipped behind a flag, but the flag was enabled globally instead of rolled out by market region.
- Checkout alerts detected the issue, but there was no specific alert for database connection saturation by service.
- The inventory reservation code retried lock conflicts immediately instead of using bounded backoff.
- The rollback checklist did not include clearing stuck reservation attempts.

## What Went Well

- Browsing, search, and vendor storefront pages remained available.
- Payments were not captured unless order creation completed successfully.
- The feature flag allowed us to disable the new workflow without a full deployment.
- Support quickly identified affected shoppers with active carts.

## What Could Have Gone Better

- We should have caught the transaction behavior in load testing.
- The rollout should have been limited to one low-traffic market first.
- Checkout error alerts fired after customers were already impacted.
- The incident channel lacked a dedicated support liaison for the first 15 minutes.

## Action Items

| Action | Owner | Priority | Due Date |
| --- | --- | --- | --- |
| Add load tests for peak Saturday traffic with multi-vendor baskets. | Platform | High | 2026-07-17 |
| Refactor inventory reservation flow to avoid holding DB transactions during vendor-level availability checks. | Marketplace | High | 2026-07-24 |
| Add bounded retry with jitter for inventory lock conflicts. | Marketplace | High | 2026-07-24 |
| Add checkout-specific database connection saturation alert. | Infrastructure | Medium | 2026-07-15 |
| Require regional rollout plans for checkout and inventory changes. | Engineering Managers | Medium | 2026-07-19 |
| Update rollback checklist to include disabling reservation workers and clearing pending reservations. | SRE | Medium | 2026-07-18 |
| Send discount credit to shoppers whose carts failed and were not retried. | Customer Operations | Low | 2026-07-12 |

## Prevention

We will keep the reservation workflow disabled until the transaction model is redesigned and verified under realistic Saturday market load. Future checkout-related releases will require regional rollout, service-level database monitoring, and a rollback plan reviewed before launch.