Hey Product Hunt! 👋 I'm Dedy, the creator of Auditi. I built this while working on AI agents in production. The hardest part wasn't building the agents — it was knowing if they were actually working well and when/where they are not performing as expected. I tried the existing tools: Langfuse (too complex), LangSmith (closed source, vendor lock-in), and various logging tools (zero evaluation). So I built Auditi. Here's what that looks like in practice: #python import auditi auditi.init() auditi.instrument() # All OpenAI/Anthropic/Google calls are now traced + evaluated 3 lines. No code changes to your existing app. Every LLM call gets traced, costed, and can be automatically evaluated for relevance, faithfulness, toxicity, and more. When LLM judges aren't enough, route traces to human reviewers. When you find great outputs, turn them into fine-tuning datasets. And everything runs on your own infrastructure — docker compose up and you own your data. I'd love your feedback, what features would make this most useful for your team?

Auditi

Open source AI agents observability and evaluation

Open source AI agents observability and evaluation