AI Observability by OpenObserve - OpenTelemetry-native observability for agents and LLMs
by•
Your agent cost $40 and took 34 seconds. But why? OpenObserve traces every agent session across models, tools, services, datastores, and user sessions so you can see exactly where time, money, and quality went. Detect loops, run online evals, and follow failures from the LLM call through your backend and database, alongside the logs, traces, and metrics from the rest of your production stack.
Replies
OpenObserve
Hi Product Hunt, Ashish here, Head of Engineering at OpenObserve.
If your AI agent got stuck in a tool-call loop right now, would you know? Would you spot it before your customer did?
We didn't.
During a live demo, our own AI SRE Agent silently stalled. No errors. No timeouts. Nothing useful in the logs.
We had to dump raw spans and debugged through them just to find the loop.
Reading raw spans isn't a debugging strategy—it’s an apology waiting to happen.
So we built AI Observability directly into OpenObserve. We wanted to see inside the black box.
Here is what it actually gives you:
Sessions map out every single turn. You see every LLM request, tool call, token count, cost, and exactly what prompt caching saved you.
Agent Graph plots your agents, tools, and models onto the exact same service map as the rest of your backend infrastructure.
Agent Behavior automatically flags the sessions that loop or fail—long before a user complains.
Online Evals let you score live sessions using any judge model (bring your own provider and key).
Annotation Queues let you turn those ugly, failed sessions into clean datasets so you can regression-test your fixes.
The best part? It's OpenTelemetry-native.
It normalizes OTel GenAI, OpenInference, OpenLLMetry, and Vercel AI SDK and many more out of the box. Nothing you’ve already wired up goes to waste, and you don’t have to ship a second copy of your data to another platform.
Our SRE Agent runs on this daily now, and it's still our harshest critic.
Point it at your own agent traces. I'd love to hear what you find.
@ashish_kolhe2 Seems very promising.
Thin on footprint and very rich on features.
OpenObserve
@jaydipsinh_narolia Thanks a lot for your kind words
Hi guys, Hengfei here, I designed this module, so let me add the part the launch page doesn't cover — what we deliberately chose not to build.
Sessions, not calls. Most LLM tracing anchors on a single request. But agents don't fail at a call — they fail across a path: right answer, wrong tool, fourteen times. So the session is the first-class object, and cost, tokens and scores roll up to it. Spans are the substrate, not the unit of analysis.
One data layer. We could have shipped a separate AI observability product. We didn't — because half of what kills an agent isn't the model. It's a vector DB timing out, a 429 from a downstream service, a retry storm in your own API. If agent spans live in a different system than your infra traces, you get to debug the same incident twice.
Scores are append-only. An evaluation is data, not a label. Change your judge prompt and the old scores don't get overwritten — they get a new version. Otherwise "quality improved" is unfalsifiable.
Bring your own judge. The judge model is yours, self-hosted open weights included. Evaluating production traffic shouldn't require shipping production traffic to someone else.
One thing that genuinely surprised me while building this: the OTel GenAI semconv renamed core attributes twice in two years (gen_ai.system → provider.name, events → input/output.messages). Betting on a fixed schema would have been the real mistake. The mapping layer turned out to be the feature.
What's the worst agent failure you've had to debug straight from raw spans? Collecting these — seriously.
Hey @hengfei_yang , I am mostly interested in the steps required to set this up - like how easy it is. Basically, I just want to start monitoring my prompts and understand how I am utilising the agent. How can set this up quickly without the need to understand all the advanced parts of the AI observability? Also, if there is any document/article you can share, that would be great as well! And congrats for the launch!!
Hey @subhra264 It is pretty easy to start monitor you AI works. we have one command to integrate for most AI tools. Which tool are you using? Claude Code or Codex?
@hengfei_yang I mostly use Claude Code only. But I might want to switch to Codex soon. So, if you can share for both that would be great ;)
@subhra264 Both are covered. Quick start is here — about 2 minutes: https://openobserve.ai/docs/getting-started/
Once it's running, the integration guide lives inside the product: http://localhost:5080/web/ingestion/ai-integrations/popular/claude-code
gives you the one-line command for Claude Code, and Codex, OpenCode and the rest are in the same section. Paste it and your sessions, prompts and token usage start showing up — nothing else to configure. Evals and annotation queues are there when you want them, but you don't need any of that for this.
Ping me if you hit anything odd. And once you're in, tell me what you look at first — cost, or the tool calls?
@hengfei_yang hey, how does the mapping layer works here? Also can we use some custom attributes that are attached to traces in this flow? Thanks :)
@yashodhan_joshi Mapping happens at ingest. Extractors read each span and normalize the different dialects — OTel GenAI, OpenInference, OpenLLMetry, Vercel AI SDK, Langfuse and a few more — into one set of gen_ai.* fields: model, provider, input/output/cache tokens, cost, tool, agent, session and user id. It's additive, not destructive: if you already set a field yourself, yours wins — we only fill in what's missing.
And yes, custom attributes work. Every span attribute is flattened into a top-level field and is queryable and groupable in SQL like any other — nothing is dropped. If a name would collide with a reserved field it gets an attr_ prefix instead of overwriting it. session.id is just an attribute too, so session grouping is yours to control.
What are you attaching — tenant, feature flag, prompt version?
@hengfei_yang
I'm planning to send Claude Code or Claude desktop telemetry to OpenObserve. Does the OTel exporter add some lag to responses, or export run entirely in the background?
@nikhil_sai4 Claude Code uses the OTel SDK's batching exporters, so nothing ships on the request path: metrics flush every 60s by default, logs/events every 5s, and traces every 5s behind the beta flag. Exporter failures surface as errors instead of touching the session, so even if the backend is slow or down, Claude Code just carries on.
Are you doing this for one machine or rolling it out across a team? Happy to share the config we run internally.
@hengfei_yang currently I am doing this for one machine , it would be helpful if you can share the config .
@nikhil_sai4 Here's the whole thing for a single machine — drop it in your shell profile:
Two things that bite people: no trailing slash on the endpoint, and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 is what turns traces on — without it you get metrics and events but no session view.
Full doc: https://openobserve.ai/docs/integration/ai/claude-code-tracing/
Start a session and it should show up within a few seconds. Come back and tell me what surprised you most — for most people it's how much of their token spend turns out to be cached input.
@hengfei_yang the agents are loop detection sounds really very useful, especially when there is no obvious errors,
I like that this work on with Open telemetry instead of a need another data pipeline.
How do you decide which fail sessions are worth turning into a eval data?
@amrita_prajapati2 Honest answer: most of them aren't. Three filters. Can you state what the right answer should have been? If not, it's a bug report, not eval data. Does it represent a class of failure rather than one weird input? And is the failure actually in the model or the prompt — a session that failed because a downstream service timed out is an infra fix, and putting it in your eval set just teaches you noise.
In the product, flagged sessions land in a review queue, a human grades them, and only reviewed items get promoted into a versioned dataset. The human step is the point: the queue collects candidates, it doesn't decide.
How big is your eval set today?
OpenObserve
@hengfei_yang @amrita_prajapati2
Thanks! Loop detection matters because the agent isn't crashing, it's just burning tokens in circles, so nothing shows up in your error logs. And OTel-native was a hard requirement. If you already have traces flowing, you shouldn't need a second pipeline.
On evals: we don't turn every failure into eval data. Scorers flag the sessions that fall below your quality bar or trip a signal like a loop. Those go to a review queue where a human annotates what went wrong, and the ones that repeat or matter get promoted into a dataset. That dataset becomes the regression suite you run every time you change a prompt, model, or tool.
OpenObserve
@amrita_prajapati2 Thanks! And yes, I think the key is not to turn every failed session into eval data. I’d prioritize sessions that reveal a repeatable or meaningful failure pattern agent loops, unexpected tool calls, poor outputs, high cost/latency, or failures that traditional error monitoring doesn’t catch.
Those traces can then be promoted into datasets and used as evaluation cases, so the production failure becomes a regression test for future changes. OpenObserve supports promoting captured traces into datasets and running experiments against versioned datasets.
OpenObserve
@amrita_prajapati2 You can refer to the docs here: https://openobserve.ai/docs/integration/ai/llm-experiments/
OpenObserve
@hengfei_yang You might have answered that but can I compare the quality of one version for another?
@shohams Yes — three ways, depending on how confident you need to be.
Side by side on a single input, when you're still iterating on wording. Across a dataset, where both versions run over the same set and get scored by the same scorers — so you're comparing score distributions, not impressions. And in production: the prompt version rides along on the span, so you can slice live quality, cost and latency by version and see whether the one that won offline is still winning on real traffic.
That last one is the one people skip, and it's where offline wins usually go to die.
Are you comparing prompt versions, or model versions?
Does ai observability share the same storage and query layer as the APM/Infra monitoring engine?
OpenObserve
@vijeeta_mittal1 yes it shares same storage & query layer which is already optimised for PB scale workloads.
OpenObserve
@vijeeta_mittal1 we have rust based engine, which lets you correlate every layer of your stack.
@manas_sharma9 is the correlation happen automatically or do we have to configure some thing on our end?
OpenObserve
@vijeeta_mittal1 it does happens automatically, we have something called auto-discovery which detect what services are sending data and what dimensions can be accounted to correlate it.
What instrumentation do you use? something proprietary or standard like otel?
OpenObserve
@vijeeta_mittal1 Yep, and that’s actually one of the things I find most useful about this. AI telemetry isn’t sitting in a separate data layer, so you can correlate it with your existing application and infrastructure telemetry in the same place.
Are you currently running your AI workloads alongside your existing APM/infra stack, or are you using a separate tool for AI observability?
@simran_kumari11 We’re using a separate tool for AI observability today. If everything is in the same data layer, can I actually correlate an AI session with the underlying application traces?
OpenObserve
@simran_kumari11 @vijeeta_mittal1 yes correlation works across logs, metrics, traces, RUM etc. So you can very well check underlying traces from AI session.
OpenObserve
@vijeeta_mittal1 @ashish_kolhe2 Yeah truee! The idea is that you don’t have to jump between systems to understand what happened. You can follow the AI session into the application and infrastructure telemetry and see where the actual issue occurred.
We have heard of cases where the AI call looked fine but something downstream was actually causing the problem? Have you run into one @vijeeta_mittal1 ?
OpenObserve
@manas_sharma9 That's good to know, can you share any guides you might have to get my claude code usage data into your platform?
OpenObserve
@vijeeta_mittal1 You can refer to this: https://openobserve.ai/docs/integration/ai/claude-code-tracing/ .
@simran_kumari11 Thanks.
Cool service! Can I break everything down by user and see statistics for each user? We have many agents working within a single user session, and that would be very useful for unit economics.
OpenObserve
@natalia_iankovych Thanks! Yes. Everything comes in as OpenTelemetry traces, so if your agents tag spans with a user ID (user.id or your own attribute), you can group by it. Multiple agents in one session roll up under that session, and sessions roll up under the user.
From there you get per-user token usage, cost, latency, tool calls, and error rates across all their agents. That's the unit economics view: what each user actually costs you to serve, and which agent is driving it.
If you're not tagging user IDs yet, that's a one-line change in your instrumentation.
Curious @natalia_iankovych , are your agents sharing one model or mixing models across a session? That changes how you'd want to break cost down.
OpenObserve
@natalia_iankovych Thanks!
To add. Since the traces are OpenTelemetry-based, you can attach user/session attributes to your telemetry and use those dimensions to analyze the data.
OpenObserve
@natalia_iankovych This should work well for a multi-agent setup where several agents contribute to a single user session.
A few useful docs:
LLM & Agent Observability
LLM Applications with OpenTelemetry
AI & LLM Integrations
@natalia_iankovych Yes — user is a first-class dimension. user.id is normalized at ingest, so every span, trace and session carries it. Sessions are listed with their user, there's a top-users-by-cost view, and anything beyond that is a group-by: cost per user, tokens, sessions, steps, error rate.
Your case is actually the easy one — many agents inside a single user session all roll up into that session, so cost per user session is one number with the per-agent split underneath it.
For unit economics you usually want tenant or plan as well. Any custom attribute you attach to the trace works as a group-by the same way, so that's a one-line change on your side.
Are you costing per end user, or per tenant/plan?
OpenObserve
@natalia_iankovych When you combine that with session recording and frontend to backend tracing you get complete visibility and a single pane of glass.
OpenObserve
If an agent uses many models in one session, can I see how much costs for each model, and total cost for the session?
@huaijin_hao Good question, In OpenObserve cost lives at three levels: span, trace, session. Each span is priced from its own model, so a request fanning out into 30 calls across 4 models is already one session total, with per-turn and per-tool breakdowns underneath — mixed-model runs need nothing special. Insights then flags which sessions and traces are cost outliers, so you land on the expensive one instead of scanning.
What's your fan-out shape — one agent calling several models, or sub-agents spread across services?
OpenObserve
@hengfei_yang i use the sub-agents with different model, the more capable model for plan, and then give the execution to the sub-agent with cheaper moodels.
@huaijin_hao That's exactly the shape the rollup is built for. Every span carries its own model and its own cost, so cost-by-model is just a group-by — inside a single session, or across all of them. And agent name is a dimension too, so you can put the sub-agents side by side.
The interesting thing to watch usually isn't the planner. It's a cheap sub-agent that retries eight times and quietly costs more than the expensive model it was supposed to save you from.
One thing worth checking on your side: if the sub-agents run as separate services, make sure session.id (or gen_ai.conversation.id) is propagated — otherwise they show up as separate sessions instead of one.
Curious what your planner-vs-execution split actually comes out to😯
OpenObserve
@huaijin_hao That’s actually a great use case for this. With multiple sub-agents, I’d imagine the tricky part is getting the full picture , which model was used where, what each step cost, and how that adds up across the whole session.
OpenObserve
@huaijin_hao We roll all those calls up to the session, so you can see what the overall request cost you and then drill down into each model call. This gets pretty interesting with agents because one user request can end up making a lot more calls than you’d expect.
Is the trace approach applicable to multiple agents, or is it designed for single agent flows only?
Btw, Congratulations @hengfei_yang & team @OpenObserve ✌️🎊.
OpenObserve
@aymi_malik Thank you! It’s designed to work with multi-agent flows as well, not just single-agent setups. You can trace interactions across agents and follow the flow end to end, which makes it easier to understand where latency, errors, or unexpected behavior are coming from.
@aymi_malik Thanks! Multi-agent is the case it was built for — single-agent is just the simple end of it.
Agent name and id are first-class dimensions, so every span knows which agent produced it. Sub-agents show up as branches of the same session rather than separate runs, the agent graph shows how agents, tools and models connect, and you can put agents side by side — cost per session, steps, tool failure rate.
What are you running — one framework, or agents spread across a few services?
I love OpenObserve because of its very friendly to use with great UI. I will like to know if there is test package that can be used as a proof of concept with my team?
OpenObserve
@simon_oluyomi
Thanks, that's great to hear. Yes, a few ways to run a POC depending on how your team wants to work.
Fastest is OpenObserve Cloud. Sign up, get a free tier, and you're ingesting in minutes. No infra to stand up.
If you need it self-hosted, the open-source build is a single binary or a Helm chart, and it's the same core product. If your POC needs enterprise features like SSO or RBAC, we hand out free self-hosted enterprise license keys for evaluation. Just ask.
BTW, Which way does your team lean, cloud or self-hosted? I can point you at the right starting kit.
OpenObserve
@simon_oluyomi Hi Simon,
You could use either of
open-source self-hosted (start in under 2 minutes -
)enterprise self-hosted product, which is free for 50 GB per day ingestion forever
OpenObserve Cloud, which has a 14-day trial
All of these are great options.
@simon_oluyomi Yes — and you don't need a special package for it. OpenObserve is open source: self-host it and run the PoC on your own infrastructure, no trial clock and no license key. Install is a single binary or one docker run, a couple of minutes, and you can point real traffic at it the same day.
If your PoC has a shape to it — a volume target, a migration off something else, a specific agent workload — tell me and I'll point you at the right setup so you don't lose the first week to tuning.
OpenObserve
Hi Product Hunt, Simran here from the OpenObserve team.
One thing we kept running into while working with AI workloads: an LLM call can succeed while the agent still fails.
That’s the gap we wanted to solve with AI Observability.
Instead of looking at LLM calls in isolation, we wanted to see the entire session: LLM calls, tool calls, tokens, cost, latency, evaluations, and the infrastructure underneath it.
And because it’s built into OpenObserve, those AI traces live alongside your existing logs, metrics, and application traces. No second observability stack just for your AI workloads.
It’s also fully OpenTelemetry-native, so you can bring telemetry from the frameworks and instrumentation you’re already using.
If you’re already running AI workloads in production, I’d be curious to hear what you’re actually using today to debug them.
@simran_kumari11 Thats great! I already use OTEL for Logs and Traces.
Do I need any changes in current setup to start with AI Traces in openobserve ?
OpenObserve
@omkark If you’re already using OTEL for logs and traces, you’re pretty much halfway there. You can send the AI telemetry through the same OTEL pipeline, so you don’t need to set up a separate observability stack.
OpenObserve
@omkark You can check out the docs: https://openobserve.ai/docs/integration/ai/llm-applications/
@simran_kumari11 Thanks!
Is there any need for additional SDK or custom intrumentation for LLM calls ?
@hengfei_yang for someone who wants to configure evaluation pipeline, how easy is it going to be? I am also looking to score the responses, say hallucination, risky and etc. With the agentic workflows we have lots of tool calls and that takes a while so how could we trace these tool calls and improve them?
OpenObserve
@chaitanya_sistla That's a good usecase.
You can set up an evaluation pipeline without building a separate evaluation data pipeline. OpenObserve’s Online Evaluations lets you define scorers using an LLM-as-a-judge or a remote scoring endpoint, and apply them at the span, trace, or session level. So you could have separate scores for things like hallucination, safety/risk, quality, or whatever signals matter for your application.
OpenObserve
@chaitanya_sistla For the tool-call side, the OpenTelemetry instrumentation captures tool invocations as spans alongside LLM calls, agent steps, and retrieval, so you can open the trace and see exactly where time is being spent and which tool calls are contributing to latency or failures.
You can then use those traces as evaluation cases too ,traces can be promoted into datasets, and experiments let you run the same cases against different versions and compare whether things actually improved.
For reference: https://openobserve.ai/docs/integration/ai/llm-evaluations/
@chaitanya_sistla Two parts:
Evals — you pick a judge provider once, then switch scorers on. Groundedness/hallucination, safety, relevance and tool effectiveness ship built-in, and they run continuously against production traffic rather than only offline datasets, so you catch drift instead of just errors. Custom means a prompt (LLM judge) or a small code scorer.
Tool calls — every call is its own span with latency and result, so inside a session you get tool hotspots: which tool is slow, how often it's called, how often it fails. Across sessions you can rank tools by total time spent. That's usually where the win hides — one tool called five times when once would have done.
Which tools hurt most for you, retrieval or external APIs?
How long does it take to go from a fresh install to seeing my first agent trace? What does the instrumentation actually look like?
@charity_lee1 Fresh install to first trace is a couple of minutes.
If you're monitoring a coding agent like Claude Code or Codex, instrumentation is literally a few environment variables — endpoint, auth header, exporter flags. No code.
If it's your own app: already emitting OpenTelemetry? Point the OTLP endpoint at OpenObserve and you're done — the LLM and agent spans arrive through the pipeline you already have. If not, you add the auto-instrumentation for your framework, usually an import and one init call.
Which is your case — a coding agent, or your own app?
OpenObserve
@charity_lee1 Adding to Hengfei's answer, the "couple of minutes" is real, and the reason is that we didn't invent a new agent, SDK, or data format. It's OpenTelemetry end to end. The same OTLP endpoint that takes your logs and traces takes your agent spans.
What you should look at first is the Agent Graph. Every LLM call, tool call, and handoff in the session laid out as a trace, with tokens and latency on each step. That's usually the moment people realize how much their agent was doing that they couldn't see.
If you want, tell me your framework and I'll point you at the exact init snippet.