How do you handle usage limits when API costs start slowing down product development?
by•
My co-founder and I love how fast we can ship using OpenAI endpoints, but the unpredictable monthly bill is forcing us to cap feature usage. It feels terrible having to limit user experiments just to keep costs under control.
We're tempted to switch to self-hosted open-source models for predictable hardware costs but taking on server maintenance and potential drops in output quality gives us pause.
Did moving away from paid AI APIs help your team ship faster or did devops overhead slow you down?
3 views
Replies
It's an interesting challenge. I was recently reading an article (highly recommend this blog btw -> https://magazine.sebastianraschka.com/p/using-local-coding-agents) comparing Qwen 3.8 Max to the flagship openai/anthropic models, and it's exciting. But still, the hardware and support needed to run it locally is not trivial.
But here is an idea I actually made work recently. As part of my harness, there is a decision tree based on the work done, or being planned. It also depends whether the agent codes, reviews, architects, or tests. And depending on a certain score it will use a bit more simple/older/cheap model for these particular tasks only. Obviously, when architecting a new feature it will need to use the best model there is, but you could dynamically swtich these when there are simpler tasks.
You could also run an experiment and measure whether the amount of tokens used for tasks is optimized. Are you using unnecessary context, repeatedly using the same session window, etc.