The Payment API That Charged Customers Twice

A client's checkout endpoint had idempotency keys. Customers still got double-charged during flash sales. The bug was a three-line race condition between SELECT and INSERT that took two weeks to find.

The first support ticket said "I was charged twice for order #4781." The second one came in twenty minutes later, different customer, same complaint. By end of day, the support team had eleven double-charge reports, all from a Friday flash sale that had pushed checkout traffic to about 4x normal.

My client ran a mid-size e-commerce platform — around 60,000 orders a month, processing roughly $2.3 million through Stripe. I was three weeks into an engagement focused on API reliability and had been working through their error budget when the double charges started showing up.

The team's first reaction was reasonable: check Stripe's dashboard. And sure enough, for each affected order, there were two successful charges with two distinct payment intent IDs. Same customer, same amount, same card, within 200-400ms of each other.

"But we have idempotency keys"

That was the lead backend engineer's response, and it was fair. The checkout endpoint accepted an idempotency_key parameter, generated client-side as a hash of the cart ID and user session. The flow looked correct on paper: before creating a Stripe charge, check if a payment record with that idempotency key already exists. If it does, return the existing result. If not, proceed with the charge.

Here's a simplified version of what the code looked like:

async def process_payment(idempotency_key: str, amount: int, customer_id: str):
    existing = await db.fetch_one(
        "SELECT id, stripe_payment_id, status FROM payments WHERE idempotency_key = $1",
        idempotency_key
    )
    if existing:
        return existing
 
    stripe_intent = await stripe.payment_intents.create(
        amount=amount,
        customer=customer_id,
    )
 
    await db.execute(
        "INSERT INTO payments (idempotency_key, stripe_payment_id, amount, customer_id, status) "
        "VALUES ($1, $2, $3, $4, $5)",
        idempotency_key, stripe_intent.id, amount, customer_id, "completed"
    )
    return stripe_intent

Can you see it? I didn't, not immediately. The code reads fine. The logic is correct in a single-threaded world. But this endpoint was handling 80-120 concurrent requests during the sale.

The gap between SELECT and INSERT

The bug is a textbook TOCTOU — time-of-check to time-of-use. Two requests arrive within milliseconds carrying the same idempotency key. Both execute the SELECT. Both get back None because neither has inserted yet. Both proceed to charge Stripe. Both succeed because Stripe sees two distinct API calls (the system wasn't forwarding the idempotency key to Stripe either — a separate problem). Both insert a payment record, and since there was no unique constraint on the idempotency_key column, both inserts succeed.

The window is small. Under normal load, requests are spaced far enough apart that the first one inserts before the second one checks. But during the flash sale, the checkout button was getting hammered. Some users clicked multiple times. The frontend had a loading state, but on slower connections the button was clickable for about 300ms before the spinner appeared. That was enough.

What the logs showed

I pulled the payment records for all eleven double-charged orders and found the pattern immediately. Every pair of duplicate payments was inserted within 50-350ms of each other. The idempotency keys were identical. The created_at timestamps overlapped.

SELECT idempotency_key, count(*), 
       max(created_at) - min(created_at) as gap
FROM payments 
WHERE created_at > '2026-08-22' 
GROUP BY idempotency_key 
HAVING count(*) > 1;

Eleven rows. Gaps ranging from 47ms to 338ms. All from a two-hour window during peak traffic.

The team had never seen this before because their normal checkout rate was about 3 requests per second. During the sale, it spiked to 12-15. The probability of two identical requests hitting the gap between SELECT and INSERT is low at 3 req/s. At 15 req/s with users stress-clicking checkout, it becomes a matter of when, not if.

The fix was three lines

The real fix wasn't in the application code. It was in the schema.

ALTER TABLE payments 
ADD CONSTRAINT payments_idempotency_key_unique 
UNIQUE (idempotency_key);

Then the application code changed to use the constraint as the source of truth:

async def process_payment(idempotency_key: str, amount: int, customer_id: str):
    try:
        stripe_intent = await stripe.payment_intents.create(
            amount=amount,
            customer=customer_id,
            idempotency_key=idempotency_key,  # pass it to Stripe too
        )
        await db.execute(
            "INSERT INTO payments (idempotency_key, stripe_payment_id, amount, customer_id, status) "
            "VALUES ($1, $2, $3, $4, $5)",
            idempotency_key, stripe_intent.id, amount, customer_id, "completed"
        )
        return stripe_intent
    except UniqueViolationError:
        existing = await db.fetch_one(
            "SELECT * FROM payments WHERE idempotency_key = $1",
            idempotency_key
        )
        return existing

Two changes that matter: the database enforces uniqueness (not the application), and the idempotency key gets forwarded to Stripe so even if our code has a bug, the payment processor won't double-charge.

We also added a debounce to the checkout button and disabled it immediately on click rather than waiting for the spinner component to mount. Belt and suspenders.

The thing that bothered me

The original code had been reviewed, tested, and running for fourteen months. It had processed over 700,000 payments without a single duplicate. The team had every reason to believe idempotency was working.

And it was — until it wasn't. The race condition only manifests under concurrent load with duplicate keys, which means your test suite will never catch it unless you specifically test for concurrent submissions. Nobody writes that test because the happy path works and the code looks correct.

I've seen this pattern at three other clients now. The check-then-insert approach to idempotency shows up everywhere — not just payments, but order creation, notification dispatch, webhook processing. It works fine at low concurrency and breaks quietly at high concurrency.

If you're relying on SELECT-then-INSERT for any kind of deduplication, go check whether you have a unique constraint backing it up. Right now, before your next flash sale. The database is the only thing that can atomically prevent duplicates. Application logic can only detect them, and detection after the fact means you're already issuing refunds.

What other "works until it doesn't" patterns are hiding in code you trust?