Launching today
Tessary
Stop your agents from failing silently in production
49 followers
Stop your agents from failing silently in production
49 followers
Tessary is an open-source agent reliability platform. It monitors every production trace, detects failures that sampled evals miss, and investigates the root cause with your agent codebase. It runs cheap, narrow classifiers across every trace. When it detects a recurring issue or change in behaviour, it groups the affected traces into a finding, validates the issue, and investigates what caused it. This keeps monitoring cheap, with deeper analysis reserved for issues that warrant investigation.
Hey PH 👋, I'm Akhil, co-founder of Tessary.
About a year ago, @nina_nixes and I started building synthetic users for usability testing of B2B software. We built agents that would personify real users and use the software, but we couldn't get them to work reliably enough over long sessions.
We built a massive eval set to tune them, but then evaluating a single agent session became about 5x more expensive than actually running it because we needed to grade across its many narrow intents.
That's where Tessary originated. We wanted to catch more of these failures and understand what caused them without making evaluation prohibitively expensive.
The approach we've built uses cheap, narrow classifiers to monitor every production trace, with deeper analysis reserved for issues that warrant investigation. It's open source, and you can self-host it or try it on our free Cloud plan. Here's a quick interactive walkthrough to check it out - Demo.
I'd love to hear from others working on agent reliability. What failures are you still catching manually, and what have you had to build your own checks for?
Thanks for taking a look. @nina_nixes and I are here for questions and feedback.
Congrats on the launch! We currently use Braintrust and use scoring on trace samples for evals, how is yours different?
@agarwalpranay96 - The issue is with the scoring. Agents have many intents they need to address in their prompts and scoring needs to happen across those narrow intents. This scoring usually happens on SOTA models for best accuracy but, that makes it prohibitively costly to do it for all traces, which I'm assuming is why you sample traces as well.
We have two layers of filters, the first layer checks every trace at really cheap cost, like 2-3 orders of magnitude cheaper than SOTA. This layer then filters "potentially erroneous" traces for deeper triage and RCA to find the issue in your agent codebase. This approach actually finds the long tail of low probability issues that sampled evals miss while actually being cheaper than scoring.
That story about the evaluation bill is a pretty good reason to build this. Narrow checks on every trace, with a closer look at recurring problems, sounds like a sensible place to start.
@oleksii_sekundant - Yes, and with our approach, the evaluation will get cheaper as the agent gets more reliable, which is the way it should be. Scoring approach doesn't have this benefit, it in-fact increases in cost as volume increases.
Hey everyone! I'm Nidhi, co-founder of Tessary. I'm really looking forward to hearing from you folks about our product!
Across the many agents we've seen being deployed in production, one behaviour seems to be consistent. Builders are constantly agonized by the edge cases that go unhandled, or LLMs being pedantic with excessive instructions. Having to find these out has been painstakingly difficult, parsing through millions of traces manually, followed by writing expensive evals to prevent them.
We built Tessary to help teams avoid this mess. Metrics for agents should be as easy to track as traditional software, know exactly the kind of behaviour that goes for a toss, with data to back it up.
If you've experienced a similar pain before, we'd love to chat! Try us out for free at app.tessary.ai :)