Launched this week
three.dev
Find out how your AI agent fails. Before your users do.
215 followers
Find out how your AI agent fails. Before your users do.
215 followers
Almost every team now runs AI in production, but few know how it behaves once real users show up. The failures are quiet: a wrong refund policy, a made-up order status. Nothing crashes, so you hear about it from your users. three.dev finds those failures in your live traffic on its own and ranks them by severity. Setup is one prompt to Claude Code, Cursor, or Codex. Then your coding agent gets what it needs to write the fix.
Free Options
Launch Team
AWS Activate CreditsCredits that fuel your startup’s next milestone
Promoted
Hi Product Hunt, I'm Bryan, product lead at three.dev.
AI features can fail quietly. A support assistant tells a customer the refund window is 60 days when the policy says 30. Nothing crashes and no alert fires. You only find out when a customer complains, or when you happen to open the right request at the right time.
Most teams we've talked to have no way to see this. They read a few conversations by hand after a complaint. Evals don't close the gap, because you can only write an eval for a failure you already know about.
So three.dev starts from your live traffic:
Get started by pasting one setup prompt into your coding agent.
We score your live requests in the background, with no added latency.
Issues are found automatically and grouped into failure modes, ranked by severity, each with the real conversations and evidence behind it.
Hand the top one to your coding agent over MCP. It reads the examples and proposes a targeted fix.
I'd love to hear from anyone running AI in production:
How do you find out today when your AI gets something wrong?
What's the worst quiet failure you've caught, and how long did it run before you saw it?
For the Product Hunt community: sign up by Nov 22, send traffic within a week, and the Starter plan is free for six months. That's $474 off ($79 a month).
I'm here all day to answer questions. Try it on your own traffic and tell us what you find!
@bryan_lee the refund window example is so real. those quiet wrong answers are scarier than crashes because nothing tells you
Congrats on the launch three.dev team! How much work is the setup? How much would I have to change the code of an existing project? Also, which languages or SDKs do you support?
@irodrisa1 thanks!
For the setup: It should be a few minutes using a coding agent with the prompt we have on the app. The `three-dev-setup` skill finds your LLM calls, generate all the necessary changes (group them by feature, and generate the use cases) and wires them up. You can also do it by hand as well because the code changes are very few.
About code changes: three.dev is a proxy, so there's no SDK to install. You just need to:
- change the base URL to `gate.three.dev` (e.g., https://gate.three.dev/v1 if you are using OpenAI)
- add a few headers: your three.dev key, the provider (e.g., `openai`) and a name of your feature (your use case)
Nothing else. Turning it off is just to revert those changes, and that's it.
Finally, about languages and SDKs: as we were mentioning before, three.dev is a proxy, so basically we support every language that can make an HTTP call, because it works at HTTP level. You keep using the official OpenAI, Anthropic, or Google GenAI SDKs, or frameworks built on them like LangChain.
Dropping here a couple of good links to follow-up:
- Skills repo: https://github.com/round3ai/skills
- Our docs: https://docs.three.dev/
Happy to help if you try it!
the refund window example (60 days when the policy says 30) is exactly the kind of failure nobody catches, no error no alert, just a confident wrong answer
how do you decide an answer is wrong without the source of truth??? do you pull in the docs whcih the agent used, or is it judged from the conversation alone???
@thamibenjelloun
Yeah, a wrong refund window is one of the many cases where the agent confidently answers something that is incorrect (and the user takes the answer for granted).
The evaluation is done on the conversation and the context that the agent retrieved when responding. For example, if the agent consulted the policy with a tool call, that additional context is given to the evaluator to determine whether the agent's response was acceptable or not.
In similar customer use cases, this allowed the evaluator to detect two big failure modes related to policies:
The agent did not consult the policy, but should have (based on the prompt instructions)
The agent consulted the policy, but it responded incorrectly
I love this idea and they've implemented it great. Evals are hard, each time I've built evals for a production app they've ended up abandoned and sadly end up spending my time being reactive when the model is acting strangely or different in prod. Love what the team here has built to solve this!
This is the failure mode I actually worry about. My agents don't crash — they confidently give a wrong answer and I only find out when a user tells me. The proxy setup is what makes it realistic for me to try: no SDK, just point the base URL at gate.three.dev and add a few headers. And failure modes ranked by severity with real examples, then handing the top one to a coding agent over MCP for a fix — that's a tight loop. Curious how noisy the severity ranking gets in the first week, before it has enough traffic to know what's actually common versus a one-off?
Thanks for the question @jcbuilds!
Individual issues are grouped into failure modes only when three.dev sees multiple occurrences of them, and the severity is automatically assigned at that stage (although it can be changed by users). An issue that is a one-off is detected but it never becomes part of a failure mode, hence no severity is assigned to it. Severity also reflects how bad a failure is, not how often it happens, so it doesn’t change as traffic grows.
yeah, this.
user-facing agents have enough speed and scope to do the job... that said, there is still softness around accuracy and MTTR for accuracy-related issues. I like that this is drop-in and does that one thing very well... much better in the long run than bringing in another "platform" or trying to whitelabel the entire agent from soup to nuts.
Really useful to catch AI mistakes in real conversations before customers start reporting them