Omari Gaskins Jr.

I ve been building a fully on-device RAG app for Android, and shifting from cloud APIs to pure local compute has been a wild ride. Getting local inference to run smoothly using Vulkan GPU compute and strict CPU thread-pinning without completely melting the phone's battery is way harder than people admit.

For those of you building with Local LLMs, are you sticking to desktop/server deployments, or is anyone else braving the mobile ecosystem? What has been your biggest hurdle?

I think AI is incredible but I do think people over-engineer solutions when it's not necessary. When would you say personally that adding AI would be a detriment to your product offering, or when do you think it makes more sense to look for more deterministic solutions.

Honestly when I am in the middle of a fast prototype, reading through hundreds of lines of generated diffs kills my momentum completely.

I usually just keep hitting accept and checking the live preview in the browser.

It works great until a weird regression pops up three hours later and i have no idea which prompt caused it.