How do you handle LLM hallucinations in your pipeline?
RAG retrieves the right documents, but LLMs still make up details not present in the source data.
I kept hitting this wall, so I built Kremis. It shifts the approach from probabilistic guessing to deterministic verification.
Here is what it actually does:
1. Ingests your data to build a strict knowledge graph.
2. Evaluates every LLM query result against that graph.
3. Outputs a concrete classification for every claim: FACT, INFERENCE, or UNKNOWN.
No fuzzy confidence scores. If a claim isn't explicitly supported by the graph, the system flags it as UNKNOWN immediately.
I'm curious how other makers are handling hallucination in production. Do you:
- Post-process LLM outputs with a secondary verification step?
- Rely entirely on citations/attributions from the model?
- Accept a certain rate of hallucinations as the cost of doing business?
Would love to hear what's working (or failing) for you.
(For context, the open-source project is here: github.com/TyKolt/kremis)

Replies