General
Share and discuss tech, products, business, startups, or product recommendations
Start new thread

Is anyone else running Local LLMs on mobile? The hardware bottlenecks are brutal.

by•

I’ve been building a fully on-device RAG app for Android, and shifting from cloud APIs to pure local compute has been a wild ride. Getting local inference to run smoothly using Vulkan GPU compute and strict CPU thread-pinning—without completely melting the phone's battery—is way harder than people admit.

For those of you building with Local LLMs, are you sticking to desktop/server deployments, or is anyone else braving the mobile ecosystem? What has been your biggest hurdle?

11 views

Add a comment

Replies

Best
One of our enterprise clients does it. Honestly outside of making your own data center, it’s really hard to do efficiently.

 Exactly! Managing local LLM clusters for enterprise is an absolute infrastructure nightmare. That’s actually why I forced all the compute strictly onto the edge (the user's phone) for DocMind. If you can optimize the inference to use the local mobile GPU, you completely bypass the need for a data center. But getting that efficient enough to not drain the battery in 10 minutes... that's the real pain point!