AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.

@noctis06 
Today, the main eval suites and traces live per project in AgentX, so teams can inspect runs, compare results, and debug failures in one place.

Where we want this to go is exactly what you described: portable/versionable eval artifacts that can run as part of CI, not just live in a UI.

For provider simulation, we’re designing around flexibility: teams should be able to test across providers without being locked into one setup, whether that means BYO keys or a managed/proxied flow depending on their workflow

@alexander_gray3 Great question Alexander! We actually have a setting that allows you to choose the repetition amount for each test set. Because we also realized that, for same question, the natural of non-deterministic may lead to different answer each time with same exactly setup. So you can totally put say 10 repetition for certain test case to make sure there will be no consistency failure.

@amna9 
It's designed to be a lightweight integration. If you already have an agent, you can connect it using our official AgentX Python SDK and start evaluating it without major changes to your existing stack. For most teams, it's more of an SDK drop-in than a large integration project.

The official SDK is available via AgentX Python SDK GitHub.

@pawel_dmochewicz Yes, Paweł, tanks for this question! That's one of the core use cases. AgentX lets you simulate and evaluate your agents across multiple LLM providers, then compare performance, cost, and latency side by side. Instead of guessing which model to use, you can make data-driven decisions before deploying changes to production.

@david_vilalta Glad the framing lands - and you've put your finger on exactly why we don't do hard pass/fail on a single sample.

Three pieces handle the non-determinism:

1. Multiple runs per case. Every case has a number_of_requests setting, so we run it N times and score the distribution rather than a single shot. One unlucky sample doesn't get to define the case.

2. Graded scoring, not binary. Each response gets an LLM-as-a-judge rating (0–10), with optional cosine and Jaccard similarity (0–1) against the expected output. Those average across runs and across cases - so an off-sample nudges the average instead of flipping a red/green bit.

3. Thresholds on the aggregate. Teams define their own eval criteria and quality thresholds, and a regression has to push the averaged score past that bar to fail the run — which then blocks the deploy, the same way automated tests stop a bad release. So yes: it's a tolerance band on the distribution, not pass/fail per sample.

How are you handling it on your side today - straight averaging, or a "k out of n must pass" rule?

@new_user___10520260379921a76fc2d64 

That's exactly the gap we set out to close. Most platforms stop at "here's your agent, good luck" - and teams end up finding out if it works the way every software team used to before CI existed: in production, from users.

Build without eval is just a prototype you haven't stress-tested yet. We wanted the whole loop in one place - build, evaluate, deploy, so shipping with confidence isn't a separate project on top of building the agent.

Appreciate you putting it so cleanly.

@olivia_bennett7 Thank you! Here's what the feedback actually looks like in practice.

You get a top-level score summary first: average rating, score range, variance and consistency across all runs. Under that, the AI Analysis runs a panel of multiple LLM judges and tracks agreement between them: how close their scores were, mean absolute error, inter-judge kappa. If judges strongly disagree on a case it gets flagged for human review.

The breakdown then goes section by section: overall assessment, instruction adherence, response patterns, reasoning analysis, tool usage. Each surfaces a different failure mode rather than bundling everything into one number.

At the bottom is Suggested Instruction Changes - concrete edits to your agent's instructions based on what the analysis found. Not "something's wrong," but specifically what to fix and why. What kind of agent are you evaluating?

@olivia_bennett7 
Thank you!

For failures, the goal is to show more than “this failed.”

We want to surface where it failed, why it likely failed, and what to check next - for example wrong tool choice, bad parameters, missing context, weak instructions, dropped fields, or a downstream step issue.

Then teams can replay the case and verify the fix instead of guessing.

@barnaby_lloyd Yes - that's actually where it matters most.

Single agent evals are straightforward. With multi-agent teams the hard part is knowing which agent in the chain introduced the failure, not just that the final output was wrong. We trace both the individual agent steps and the overall workflow, so you get a per-agent breakdown rather than one opaque score at the end.

The AI analysis then surfaces where exactly the delegation broke down: wrong routing, instruction drift, tool call failure, and suggests the specific instruction changes to fix it.

What kind of multi-agent setup are you working with?

@barnaby_lloyd 
Thank you!

Yes - multi-agent systems are one of the main areas we care about.

AgentX can evaluate not only the final answer, but also how agents coordinate: handoffs, tool usage, role boundaries, intermediate steps, and whether the full team completed the intended workflow.

That’s where debugging gets especially important.

@jocky Great sense Jocky! Developers can setup the eval pipeline into their dev -> prod pipeline and use our eval score metrics to determine the pass gate.

@pradyumna6 Yes, I totally feel you. There are many cases the Agent hallucinated the output that happens to be correct. That's why our eval looks into the deep reasoning, tool use, data retrievals, etc to make the judgement, not just output.