My own test suite took my AI app down. What are you doing about free-tier daily limits?
I’ve been testing a safety check for an AI app by running a script of test messages many times against the model. Every run was fine until it wasn’t: after a day of repeated runs, calls started failing with 429 errors, and waiting 60 seconds between retries stopped helping.
I assumed the limit was per minute. It wasn’t. On the free plan for the model I use, the limits are 30 requests a minute, 1,000 requests a day, 8,000 tokens a minute and 200,000 tokens a day. I believe I hit the daily cap, because the app itself also stopped generating responses at the same time. My tests and the app share one account, so they share one budget.
What I’m changing: run tests in small batches, never repeat full runs, and plan a backup provider so a quota problem doesn’t become an outage.
How do you keep testing from eating your production quota? Separate accounts, a cheaper model for tests, or something else?
Replies
Separate keys for tests and prod is the first fix, so a test run can never take the app down again. After that, most of my test runs don't hit a paid or rate-limited API at all. I record real model responses once and replay them, so the logic around the model gets tested for free. For things that need a live model, I run a small local model with Ollama. It's worse than the real one, but good enough to catch broken prompts and parsing. The real model only runs a small final batch before shipping.