FetchSandbox
API integration testing that remembers what breaks
Start new thread

How do you test the "book a flight to London" agent workflow

by
•

How do you test a workflow that spans four services?

Been hitting the same wall for a while and curious how others handle it.

What's the failure shape?

Someone books a flight to London. Four things have to happen: confirm the booking, send a Slack message, fire a confirmation email, block the calendar. Travel is one vendor, messaging is another, email is another, calendar is another.

We ran two versions against the same request. The second sends the right message, fires the right email, confirms the booking. Every response is a 200. What it never does is block the calendar.

Nothing looks wrong. The flight is booked. The user got a message and an email. They find out at 6am when they double-booked the same slot.

Why is this hard to catch?

A version can pass the call log check, pass the write check, and still fail the only question that matters. Is that slot actually blocked? Those are four separate questions and they give different answers.

What actually helped?

Re-running from identical state mattered more than I expected. Otherwise you're comparing two runs that started differently and calling it a diff. We also needed the services to behave like prod, 429s, auth failures, delayed webhooks, duplicate deliveries, without real prod keys. That's what FetchSandbox gives us: service twins that surface the failure modes the happy path never hits.

Is anyone actually testing these flows end to end, or is it mostly per-service mocks and hope?

63 views

Add a comment

Replies

Best

Mostly per-service mocks. And the one that actually got us wasn't a vendor.

Our version is a payment webhook. Handler writes an idempotency marker, does the work, a retry that sees the marker skips. Sounds right. Our driver has no transactions, so when a step under the marker threw, the marker survived. The retry read it, logged "already handled", returned 200. The credits never landed on the account. Every response green, the one thing that mattered silently never happened, and we heard about it from the user

So the row I'd add to Anuj's fault matrix is your own dedupe layer, not the vendors: what happens when a claim gets taken and the work under it throws. Same point catches the identical-state part too. Your marker rows are state. Reset the service twins but not your own claim table and run two isn't run one, it's run one with a poisoned skip. Duplicate web hook delivery is the twin behavior that found us the most real bugs, so that part of FetchSandbox I'd actually use.

 the marker-survives-but-work-doesn't shape is nasty because the idempotency logic is doing exactly what it should, it just has no visibility into whether the thing it's protecting actually completed. and your point about resetting your own claim table is the thing most people miss when they talk about identical-state replay. if you reset the twins but leave the marker rows, run two isn't a replay, it's a poisoned skip dressed up as one. we're building the scenario spec to include first-party state as part of the reset surface, not just vendor behavior. what did you end up doing to detect the partial-write case before the user reported it?

 Honestly? We didn't. The user got there first, and after that I stopped trying to detect the state and made it impossible to be in.

The marker and the effect are now the same write. One conditional UPDATE: add the credits, stamp the checkout session id onto the same row, WHERE the stored session id is distinct from this one. A retry matches zero rows and applies nothing. There's no window where the marker exists and the credits don't, because it's one statement and the database won't half-do it.

We have one place that can't be folded like that, a nested claim taken before work that can throw. It releases in the catch before rethrowing, and if the release itself fails, that gets reported with the payment intent id attached. So the last way to strand a marker is loud instead of silent.

The catch is that this only works because the effect is a row in my own database. Your calendar hold lives in someone else's system, so there's nothing to fold the marker into, and you land where Nivy is: read the state back and treat the 200 as a hint.

 That WHERE clause doing double duty as both the idempotency gate and the write is the cleanest version of this I've seen. Collapsing the check-then-act into one atomic UPDATE removes the window entirely rather than trying to shrink it. I spend a lot of time in webhook retry failure scenarios and this is the pattern that actually holds up under concurrent retries.

ours is publishing not booking but same shape. api says 200 and the post just isnt on the profile. the only check that ever caught it was reading the profile back after, not the response.

retry is worse for us tho, a duplicate post is public. so we treat the read-back as the source of truth and the 200 as a hint.

 the read-back as source of truth is exactly right, but some platforms add another layer of pain: eventual consistency means the post might not appear in the profile fetch for a few seconds even after it's actually there. so you get false negatives on the read-back too. have you had to add a delay + retry on the read side as well?