# Postmortem: Checkout Outage During Saturday Market **Incident date:** July 25, 2026 **Duration:** 9:07–10:41 AM EDT **Severity:** SEV-1 **Services affected:** Web and mobile checkout, vendor order management **Status:** Resolved ## Summary On July 25, customers were unable to complete purchases during the busiest period of our Saturday farmers-market window. A database connection surge caused the checkout API to time out or return errors. Vendors also experienced delays loading new orders. The incident lasted 94 minutes. Existing orders and payment data remained intact, and no duplicate charges were identified. ## Customer Impact Between 9:07 and 10:41 AM EDT: - Approximately 3,800 customers encountered checkout errors. - 2,146 checkout attempts failed or timed out. - 612 customers successfully completed purchases after retrying. - An estimated 1,050 orders were abandoned. - 74 vendors experienced delayed order notifications. - Order-management pages loaded slowly or failed intermittently. Customers could browse products and update their carts throughout the incident. Payment authorizations were not created for requests that failed before reaching the payment provider. We reconciled all successful authorizations against orders and found no orphaned or duplicate charges. ## Timeline All times are in EDT. - **8:55 AM:** Scheduled Saturday traffic increase begins. - **9:02 AM:** A recently deployed inventory feature starts issuing additional database queries for every checkout item. - **9:07 AM:** Checkout latency exceeds the service-level objective. - **9:09 AM:** Database connection usage reaches 95%. - **9:11 AM:** Automated alerts notify the on-call engineer of elevated checkout errors. - **9:15 AM:** The incident is declared SEV-1, and the engineering incident channel is opened. - **9:21 AM:** The team identifies database connection exhaustion in the checkout service. - **9:28 AM:** Application replicas are scaled down to reduce aggregate connection demand. Error rates decline briefly but remain elevated. - **9:36 AM:** Engineers correlate the connection increase with the inventory-reservation deployment completed the previous evening. - **9:44 AM:** The team begins rolling back the inventory feature. - **9:57 AM:** Rollback completes, but several application instances retain saturated connection pools. - **10:05 AM:** Checkout instances are restarted in batches. - **10:16 AM:** Successful checkout volume begins recovering. - **10:28 AM:** Error rates return below the alert threshold. - **10:41 AM:** Database connections and checkout latency remain stable for 15 minutes. The incident is resolved. - **12:20 PM:** Payment and order reconciliation confirms no duplicate charges or lost successful orders. ## Root Cause The outage was caused by a change to inventory reservation logic deployed the evening before the incident. The new implementation checked product availability separately for every line item and opened a database transaction before calling the external payment service. Each transaction held its database connection until the payment request completed. Under normal weekday traffic, the additional connection time did not exceed capacity. During the Saturday morning traffic peak, concurrent checkouts increased sharply. Slow responses from the payment provider extended transaction duration, causing checkout instances to retain connections longer than expected. Because each application replica maintained an independent connection pool, autoscaling added more potential database connections. The combined demand exhausted the database connection limit. Once the connection limit was reached, new checkout requests waited for connections and eventually timed out. Vendor order-management requests used the same database cluster and were affected by the resulting contention. ## Contributing Factors - Load tests modeled average traffic but not the Saturday opening surge. - The checkout service’s connection-pool size was configured per replica without a global connection budget. - Autoscaling used CPU utilization, which remained low while requests waited for database connections. - External payment calls occurred inside database transactions. - The deployment was completed outside peak hours, but no production canary was observed during a representative traffic period. - Alerts identified checkout errors and database saturation but did not identify long-lived transactions directly. - Checkout and vendor order-management workloads shared the same database connection capacity. ## Resolution and Recovery We rolled back the inventory-reservation change and restarted checkout instances in batches to clear saturated connection pools. We temporarily reduced the number of checkout replicas to keep total possible connections below the database limit. After recovery, we reconciled marketplace orders with payment-provider records and verified that every successful payment had a corresponding order. Customer support contacted customers who reported uncertain payment status and provided vendors with updated fulfillment exports. ## What Went Well - Automated alerts detected the issue within four minutes. - The on-call engineer declared a SEV-1 quickly and involved database and payments engineers. - The deployment rollback was available and did not require a schema rollback. - Payment idempotency keys prevented duplicate charges during customer retries. - Order and payment reconciliation tooling provided a definitive integrity check. ## What Went Poorly - The initial mitigation—scaling application replicas—briefly increased database connection pressure. - Dashboards did not show transaction age by endpoint, delaying identification of the inventory change. - Load testing did not represent peak concurrency or payment-provider latency. - The incident status page was not updated until 38 minutes after customer impact began. - Vendor support did not receive a prepared message for order-notification delays. ## Action Items | Action | Owner | Priority | Due date | |---|---|---:|---:| | Move payment-provider calls outside database transactions | Checkout Team | P0 | August 7, 2026 | | Replace per-item availability queries with a single batched query | Inventory Team | P0 | August 7, 2026 | | Enforce a global database connection budget across checkout replicas | Platform Team | P0 | August 14, 2026 | | Add transaction-duration and connection-wait alerts by endpoint | Observability Team | P1 | August 14, 2026 | | Add database saturation signals to checkout autoscaling policies | Platform Team | P1 | August 21, 2026 | | Create a Saturday-peak load-test profile with simulated payment latency | Performance Team | P1 | August 21, 2026 | | Require peak-period canaries for checkout and inventory changes | Release Engineering | P1 | August 14, 2026 | | Isolate vendor read traffic from checkout connection pools | Marketplace Team | P2 | September 4, 2026 | | Automate incident status-page updates from SEV-1 declarations | Reliability Team | P2 | September 4, 2026 | | Prepare customer- and vendor-support templates for checkout incidents | Support Operations | P2 | August 14, 2026 | ## Prevention The primary prevention work is to shorten database transactions and cap total connection usage independently of replica count. We are also updating load tests to model the marketplace’s bursty, time-bound traffic pattern rather than relying on daily averages. These changes will be verified in staging under projected Saturday peak load and then released through a production canary with database connection usage, transaction duration, checkout success rate, and payment latency monitored as release gates.