How do you keep an optional AI call from breaking the main one?
I’m adding a short plan (a small challenge plus two or three steps) to the results page of an AI app. The endpoint that produces the result already chains several model calls: a safety check, the main generation, and two small side calls (a share caption and a memory note), all inside a 60-second function limit. Before adding a fifth call, I had Claude Code read the route and report how close it was to that limit.
The finding was uncomfortable. The two side calls had no timeout of their own. The SDK’s default is one minute with two automatic retries, so one hung side call could run past the 60 seconds, and the platform would kill the whole request before the main result was saved. A code comment even claimed each call had its own timeout. It didn’t.
What I changed: both side calls now get a 10-second timeout and no retries, and their failures are still swallowed as before. The new plan call goes through a helper that makes one attempt with an 8-second timeout and never throws. On a timeout, bad JSON, over-long text or a failed safety check it returns nothing, and the result is saved without a plan. It starts only after every safety gate, at the same moment as the side calls, so it adds little waiting time.
A test reads the route’s source and checks that the plan call sits before the single database insert and that the side calls carry a timeout and zero retries, so a later edit can’t quietly undo it.
The honest weak spot: the worst case is now bounded under 60 seconds, but not by a wide margin on a bad day, because one helper can still make two 10-second attempts. And an optional call that gives up under load means the feature disappears exactly when traffic is highest.
How do you cap latency across chained model calls: a timeout per call, one overall deadline for the request, or both? And how do you decide which calls are allowed to fail silently?
Replies
Be the first to reply
Have a question or a thought to share? Add a comment above to start the conversation.