Looking at monthly API invoices self hosting models like Llama feels like the obvious fix for capping expenses. But diving into setup reveals unexpected devops overhead, ongoing hardware management and model optimization time.
It makes me wonder if saving on token costs is worth the trade-off in focus especially if quality takes a hit.
For teams running their own model infrastructure what specific scale made the jump worth the extra effort?
I have build close 2-3 applications till now, none succeeded. But, while starting building or even starting to market, I never thought of making it big enough. It is simply because I was working on small problems and creating small solutions for it and to be honest, never wanted to raise investments.
But, in the startup and saas scenario, there are companies and saas with minimal idea have taken investments. For me, investments are a way with which you come under pressure of increasing your numbes faster. While indie saas owners can benefit all the profits themselves while never have to be under pressure when app or startup is down etc.
What do you think? Raising investment is a good choice for founders or when is investment needed?
When mass layoffs started in tech, many people suggested that:
The layoffs were happening because, during COVID, companies hired too many people for online and remote roles.
That AI was attacking jobs.
And I still keep seeing statements from creators of various AI tools saying: No, AI won t replace you. Employees will just have time for more meaningful tasks in a company.
Using AI assistants has definitely helped our team ship new features faster over the past few months. however, the volume of generated code in pull requests has nearly doubled, making manual code reviews feel overwhelming.
since the code looks visually clean on the surface, reviewers sometimes skim through subtle logic issues that used to get caught during standard manual reviews.
how do engineering leads maintain strict code review quality when most of the PR contains auto-generated snippets?
When you ask an AI to write tests for a complex module it always defaults to the cleanest, most basic inputs. It never thinks about null values, weird timeouts or race conditions unless you hold its hand through every single scenario.
By the time I write out all the strict edge cases myself it almost feels faster than letting the agent do it.
Looking at the raw per-token math renting an A10G or L4 on AWS/Vultr looks way cheaper than paying per token to OpenAI at scale. What I'm worried about is the hidden time sink vLLM crashes prompt tuning across model updates latency optimization load balancing and keeping up with new quantization releases. As a solo founder my time is already stretched thin. Did switching to self-hosted models actually save you net money or did the extra dev/maintenance hours wipe out any hosting savings?
I'm currently building in Social tech with no plan on integrating AI in the near future. It's an app that'll enable users to connect with others near them and attend/host events.
Are you also building a product that doesn't have an AI component? How has it been? Has the available funding decreased for "unsexy" ideas since the advent of AI?
Last week I attended Tech Week for the first time in my city and had a great time!
I was surprised by how much value came from conversations that weren't planned. It feels that in a world accelerated and misconstrued by technology, the best way to establish trust is often to meet face-to-face.
For those who regularly attend Tech Weeks, startup events, or hackathons:
