How do you evaluate a classifier that gives different answers on the same input?

I’m running a 3-level classifier (0 normal, 1 distress, 2 crisis) in front of a text generator. Same model, same prompt, same message, and I still get different answers across runs. One test message scored 8 of 10 correct on my latest code and 9 of 10 on a commit from two days earlier. That difference is noise, not a regression, but a single run would have blamed my last prompt change.

What I do about it:

1. Measure the classifier alone, repeated 3 to 10 times per message, separately from the full pipeline.

2. Never trust one signal. The final level is the max of three: a regex phrase filter (it can only raise a level), the classifier, and a level the generation call itself returns. If the classifier errors and no phrase matches, the request fails closed.

final = max(rule, classify, inCall)

if (classify == null && rule == 0) return UNAVAILABLE

The signals do disagree. In one run, a message was caught only by the generation call’s own level, while the classifier-only runs caught it 3 of 3. On another message, two paths gave Level 1 from the classifier and a third gave Level 2 from the in-call level.

Costs on a free tier (gpt-oss-120b on Groq, from my laptop): classify median about 0.5 to 0.8 s, one call at 7.7 s. The pipeline went from 5 to 6 sequential calls, and my own repeated evals used up the daily token cap, which took the app down.

Open question for people who’ve done this: how many repeats do you run before you trust a result, and do you gate on a threshold, a confidence interval, or just take the max of signals and accept some false alarms?