how are you testing prompt changes before they hit prod
by•
agents made prompt edits scary for me. tweak one system prompt and a tool call 3 steps later starts failing. what helped: prompts live in git, every change runs against a small fixed set of real traces before shipping, and the last good version is always one revert away. curious what others do here, do you run evals on every prompt change or only the big ones
25 views
Replies
Small and per step, not per pipeline. My backend chains a few model calls: one writes the lyrics, one aligns them to the sung audio, one rewrites a prompt the music engine refused. When I change one of those prompts I call that step alone on saved inputs, for example the aligner on an existing song's audio, and run it several times, because one good answer proves nothing. I count it as "3 of 3 kept the name" rather than "looks fine". The full generation is expensive and slow, so it only runs once at the end. The other habit: when a prompt change fixes a bug, the case that broke it becomes one of the saved inputs.
Same idea as the reply above for the judgment part, but I would shrink what the prompt has to do in the first place.
In the NextLang (flashcard tool) I build, the model only returns a JSON array of cards. Quoting, separators, dedupe and the count cap all live in code, so a prompt edit can't break them and they get normal unit tests. One run came back with 31 cards when we asked for 30, three times in a row. No prompt wording fixed it reliably, a `limit` in the parser did.
What's left for evals is the stuff only a model can get wrong. We grade real outputs with small checks (count, duplicate fronts, wrong-language text in a field), not "looks fine".
However, where I do not do this: it's not in CI. Real model calls cost money, so I run it by hand when I touch a prompt. Which means a regression can sit there until the next time I do.