How do you set latency targets for voice UX?

We were looking at some numbers on voice interface latency this week and got into a discussion that had no clean answer. The general folklore is that 100 ms feels instant, 200-300 ms feels responsive, and anything past 500 ms starts to feel like the system is thinking. But those numbers come from generic UX research, not voice specifically.

Our observation is that voice is more forgiving than tap interfaces at short delays (people expect some processing time between "hey" and a response) but much less forgiving at long delays. 500ms feels like the system heard you but is stuck, which is worse than a dropped tap.

We do not have a real answer for where the threshold sits. It probably varies by use case. Something like a smart home command has different tolerance from a real-time transcription overlay.

Curious what numbers other teams have landed on, and how you measure this internally. Especially interested from people who have shipped voice UX at scale, not just demo apps.