I audited an AI-written billing service this week. 22 findings, 10 critical.

None of them crashed. All returned 200. Four that stuck with me: an auth check that silently lets anyone in when one env var is missing, SQL injection in the one query that wasn't parameterized, a broken permission boundary so anyone can attach a charge to a workspace that isn't theirs, and a Stripe retry that double-grants seats because there's no idempotency check anywhere.

247 upvotes, 34 comments, finished second on a Sunday. The July launch got 362 and finished third. Better rank, smaller day. Both true.

The comments were worth more than the votes. A few things I didn't have good answers for, which is more useful than an upvote.

Last launch ended with 91 comments. I answered every one, and in most of them I said "not yet, that's next."

Six weeks on, here's what actually happened.

Done: @maschiojv asked about Paddle, it now has 21 curated failure patterns instead of a bare spec import.

Most AI tools, when they're unsure, guess confidently.

That's how you burn an hour chasing the wrong bug.

We went the other way. Point ours at your repo and say what's breaking "customers are getting charged twice." Instead of guessing which API, it reads your code:

• Sees you have both Stripe and Paddle installed.

• Reads both webhook handlers your Paddle one has no idempotency check (line 16); Stripe dedupes fine.

• Tells you: "it's your Paddle handler." A fact from your code, not a hunch.

since the last launch, the goal was one thing: make FetchSandbox something you can actually gate a release on. not "looks tested" but provably tested.

what changed:

i build dev tools solo and use claude/cursor daily for API integration work. the pattern i keep seeing: agent finds a bug, fixes the code, declares done. what it does not do is rerun the exact failure scenario and prove the fix holds. that verification step was always on me.
we just shipped something in FetchSandbox to close that loop. after it catches a bug, it now nudges the agent to rerun the same request and surface a behavioral comparison, before vs after. same scenario, does it pass clean? that receipt is now automatic.
my rule: finding the bug is table stakes. proof that the behavior changed is the job. "i fixed the code" and "the fix works" are not the same claim.
curious how others handle this. do your agents ever rerun failure scenarios to verify, or does that step still fall back to you?

stripe sends webhooks at-least-once. so what's the actual risk?

one paid invoice, two deliveries, non-idempotent handler: you've just provisioned twice. you won't catch it in review because the code is correct on the first delivery. the bug only shows on the second.

so how do you even reproduce it?

I stopped reading and started running. took a real express and postgres billing app, replayed the scenario in a sandbox: same stripe webhook, delivered twice, like a real retry. no prod involved. the buggy handler provisioned seats 1, then 2, then 3. signature verification was clean throughout.

Hey all ahead of our Aug 2 relaunch, I'd rather be useful than hypey. Here's the honest map: what we test, what we don't (yet), what languages it works with, and where we're going.

The failures we test for (the "it passed, then broke in prod" stuff):

• Double-processing the same webhook arrives twice; do you charge the customer twice?

• Out-of-order / late events a "canceled" and a stale "active" land in the wrong order; does a canceled account come back to life? (live for Paddle today)

• Sad paths card declined, payment failed, thing-not-found; does your code handle it, or quietly mark it "done"?

• Rate limits the API throttles you (429); do you back off, or silently drop data?

• Wrong-shape responses the API returns a field you didn't expect or renamed; does your code break downstream?

• Auth slips bad/expired key, or a read-only key doing writes; caught or not?

• Fake / replayed webhooks unsigned, tampered, or old signed events; rejected?

• Slow + flaky delivery delays and dropped webhooks.

Every run hands you a receipt: the failure reproduced, then gone after the fix.

Small milestone for FetchSandbox: we crossed 1,600+ MCP installs.

The idea started with one simple command:

./fetchsandbox verify stripe

Instead of stopping at "the API returned 200," FetchSandbox runs the actual integration flow webhooks, retries, state changes, and failure scenarios and shows what breaks before you ship.

Hey folks

If you're wiring up an API integration Stripe, Descope, Twilio, Resend, anything drop it below with what you're building.

I'll reply with the specific ways it breaks after the happy path works: the webhook that fires twice, the event that lands late, the token that's never actually verified, the state that goes stale. The stuff that passes every test and still pages you at 2am.

That's what we've spent years (and a lot of production incidents) learning and what FetchSandbox encodes so your coding agent catches it before prod.