How to check if a tuned model is stable without spending a lot of money on token bills

by•

Doing full validation checks on every checkpoint after training a language model can get really expensive fast. We used up half of our cloud budget month just checking different epoch weights on full benchmark tests. The idea that helped us save our remaining credits was creating a local test that focuses on tricky prompts. Of running thousands of validation examples we picked fifty hard prompts that show problems in our specific area. Running this test script on our own computer takes less than three minutes, for each checkpoint and catches problems right away. What method do you use to make sure your model testing costs stay low while you are training?

2 views

Add a comment

Replies

Be the first to reply

Have a question or a thought to share? Add a comment above to start the conversation.