Is the engineering overhead of self-hosting open-source LLMs worth the API savings?
by•
Looking at the raw per-token math renting an A10G or L4 on AWS/Vultr looks way cheaper than paying per token to OpenAI at scale. What I'm worried about is the hidden time sink vLLM crashes prompt tuning across model updates latency optimization load balancing and keeping up with new quantization releases. As a solo founder my time is already stretched thin. Did switching to self-hosted models actually save you net money or did the extra dev/maintenance hours wipe out any hosting savings?
7 views
Replies
Also, are you taking into account the initial hardware investment to run these? Maybe that could easily match a year of using cloud models at 20x.