Is adding AI making your product genuinely better, or just more noisy?

We recently explored adding an AI-assisted workflow to our core product. In testing, half of our users loved the speed boost, while the other half felt it added unexpected outputs and removed their control over the final result.

It opened up a bigger debate for us: Are we adding AI because it solves a real pain point, or just because users expect every modern tool to have it?

How are you setting guardrails for AI features? How do you ensure it adds real leverage rather than scope drift?