Vaishak Bangera

I ve been building a fully on-device RAG app for Android, and shifting from cloud APIs to pure local compute has been a wild ride. Getting local inference to run smoothly using Vulkan GPU compute and strict CPU thread-pinning without completely melting the phone's battery is way harder than people admit.

For those of you building with Local LLMs, are you sticking to desktop/server deployments, or is anyone else braving the mobile ecosystem? What has been your biggest hurdle?

I think AI is incredible but I do think people over-engineer solutions when it's not necessary. When would you say personally that adding AI would be a detriment to your product offering, or when do you think it makes more sense to look for more deterministic solutions.

I ve been experimenting heavily with Local LLMs and on-device RAG lately. Don't get me wrong, cloud APIs are undeniably powerful, but local models have gotten surprisingly good. The reality is that 90% of people using LLMs are just asking them simple questions massive, expensive cloud models are often complete overkill for everyday tasks.

Because of this, it feels like the hype around just wrapping the OpenAI API is dying down, especially for apps that handle sensitive data like NDAs or private client files. I recently managed to get a local RAG pipeline running smoothly on mobile using Vulkan GPU compute, and the privacy benefits are huge.

Do you think local inference will become the new standard for privacy-first apps, or will cloud APIs always dominate?