BotGauge - The trust layer for your agents
by•
AI agents don't just generate answers anymore. They take actions, call tools, and access real data. BotGauge helps engineering teams find where agents break before users do. Run adaptive red-team attacks and evals, catch security and behavioral failures, and replay every issue with full traces. Turn failures into regression tests, continuously retest as your agents evolve, and monitor their behavior in production. One reliability layer from development to deployment.
Replies
This looks really interesting. I've been hesitant to hook up some services to agents precisely because I didn't trust it not to take advantage of the permissions it was given. I'm definitely going to explore BotGauge further.
A bit of feedback on the post: your images are too low-resolution to really see what's on the screens you're sharing. I noticed the same thing on your website. I think some higher-resolution images would go a long way in helping people better understand what it is you're offering.
@nicholas_zakas Thank you, and that hesitation is exactly who we built this for. Being careful about giving an agent real permissions is the right instinct. Appreciate the note on the images too. We’ll take another look at how they’re showing up.
@pradeepsoundararajan Thank you so much! 🙏
Really love this. It's something we will definitely look into as we roll out AI chat in our app in the coming months, as it's exactly the type of testing and safeguards we need.
@billmaroulis Thank you, that’s great to hear! Rolling out chat is the perfect time to test. Sign up and try it on your app’s chat before launch, and tell us what it finds.
The part that stuck with me is turning each failure into a permanent eval that runs on every release. The quiet failures Pramin described — an agent retrying a failed tool call in a loop, or holding an approval for 117 role-play attempts and folding on the 118th — are exactly the ones that never show up as crashes. Replaying the same tool calls with the full trace feels like the only way to know a fix actually fixed it, rather than just changing the prompt and hoping. Curious how you handle the cases where the break comes from instructions hidden in a document the agent reads, rather than from the user directly.
@jcbuilds Thanks, and that 117-then-118 example is exactly why a single pass means so little. For hidden instructions in documents, we treat anything the agent reads (files, web pages, tool outputs) as an attack surface, not just what the user types. The break gets saved with the full trace, so you can replay the same poisoned document against the fix and keep it as a permanent eval.
How do I get my agent layer into the platform? Mine needs a macOS computer to run.
@alchemist_studio Good question. Can you tell us a bit more about how your agent runs, like whether it’s driven through an API, a CLI, or a desktop app? That’ll tell us the best way to hook it up, and we’ll walk you through it.
One aspect of this approach that I appreciate is that it doesn’t just focus on whether an AI agent provides the correct answer. It also sheds light on unexpected behavior. It is possible for an agent to appear to be functioning perfectly while in reality, it is making incorrect decisions. Being able to identify such edge cases early on my view extremely valuable.
Detecting errors made by AI agents can be significantly more difficult than identifying bugs in traditional software. I appreciate that, rather than treating each failure as an isolated incident. This approach integrates reusable evaluation methods with red team testing.
@tanjum Thank you, that’s exactly the idea. Agent failures don’t throw errors, they just quietly do the wrong thing, so a one-off fix doesn’t mean much. Turning each failure into a reusable test is what keeps it from coming back
Love the idea of turning every failure into a permanent test. This could save developers a lot of time and help make AI agents more reliable over time.
Replaying a failed run with full traces and then keeping it as a regression test is the loop most agent teams still do by hand in a notebook, so having it in one layer from dev to prod saves real time.
@karimbenkeroum Yeah, the notebook version works right up until the person who wrote it leaves or the agent changes underneath it. Having the failed run turn straight into a regression test means the thing that broke once keeps getting checked, and nobody has to remember to do it.
Red-teaming agents is table stakes now. One gap to explore: tool hallucination under pressure. When an agent can't solve a task, does it invent tool calls? Testing that boundary - especially with LLMs that confabulate - could catch a real production blindspot.
@omri_ben_shoham1 Yes, that’s covered. We test what an agent does when it’s stuck, including inventing a tool that doesn’t exist or faking a call result instead of admitting it can’t finish. Any break gets saved with the full trace, so you can replay it against the fix and keep it as a permanent eval.