Launched this week
BotGauge
The trust layer for your agents
678 followers
The trust layer for your agents
678 followers
AI agents don't just generate answers anymore. They take actions, call tools, and access real data. BotGauge helps engineering teams find where agents break before users do. Run adaptive red-team attacks and evals, catch security and behavioral failures, and replay every issue with full traces. Turn failures into regression tests, continuously retest as your agents evolve, and monitor their behavior in production. One reliability layer from development to deployment.
Detecting errors made by AI agents can be significantly more difficult than identifying bugs in traditional software. I appreciate that, rather than treating each failure as an isolated incident. This approach integrates reusable evaluation methods with red team testing.
@tanjum Thank you, that’s exactly the idea. Agent failures don’t throw errors, they just quietly do the wrong thing, so a one-off fix doesn’t mean much. Turning each failure into a reusable test is what keeps it from coming back
@pradeepsoundararajan Thank you so much! 🙏
One aspect of this approach that I appreciate is that it doesn’t just focus on whether an AI agent provides the correct answer. It also sheds light on unexpected behavior. It is possible for an agent to appear to be functioning perfectly while in reality, it is making incorrect decisions. Being able to identify such edge cases early on my view extremely valuable.
Love the idea of turning every failure into a permanent test. This could save developers a lot of time and help make AI agents more reliable over time.
An agent passing yesterday's test is a low bar if today's PDF can talk it into a new career. Do your evals cover instructions hidden in documents and tool outputs, or mainly direct user prompts?
@christian__schneider Yes, indirect injection is covered. It goes after instructions hidden in documents, web pages, and tool outputs, not just direct prompts. Try it on one of your agents and see what it finds.
Red-teaming agents is table stakes now. One gap to explore: tool hallucination under pressure. When an agent can't solve a task, does it invent tool calls? Testing that boundary - especially with LLMs that confabulate - could catch a real production blindspot.
@omri_ben_shoham1 Yes, that’s covered. We test what an agent does when it’s stuck, including inventing a tool that doesn’t exist or faking a call result instead of admitting it can’t finish. Any break gets saved with the full trace, so you can replay it against the fix and keep it as a permanent eval.
Replaying a failed run with full traces and then keeping it as a regression test is the loop most agent teams still do by hand in a notebook, so having it in one layer from dev to prod saves real time.
@karimbenkeroum Yeah, the notebook version works right up until the person who wrote it leaves or the agent changes underneath it. Having the failed run turn straight into a regression test means the thing that broke once keeps getting checked, and nobody has to remember to do it.