How we reduced AI support hallucinations by restricting our knowledge base to solved tickets
During our beta we linked our AI support bot to the Notion workspace, which had old product specs and meeting notes. We believed that giving the AI support bot context would let it answer any question but the AI support bot began to create features that we had cancelled six months before.
We solved the problem by disconnecting the wiki and feeding the AI support bot only verified, resolved customer support threads from the last ninety days. We also set a retrieval threshold so that the AI support bot refuses to answer when the similarity score is lower than 0.82 and then sends the user to a human instead.
Within two weeks wrong answers dropped to zero and our support team saved about twelve hours each week because they no longer had to apologize for inaccurate AI support bot replies.
Curating clean verified data matters a lot more, than letting your AI see everything you have ever written. How do you stop outdated documents from polluting the answers of your AI support bot?
Replies
That is a good catch. How did you figure it out?