General
Share and discuss tech, products, business, startups, or product recommendations
Start new thread

Anyone else building AI features without relying on external LLM APIs?

by•

Hi all, there's a distinction I keep thinking about: apps built with AI (Cursor, Claude, Copilot writing the code) vs apps that depend on AI APIs at runtime to function.

I'm in the first camp. I used AI-assisted coding heavily to build Briskly (a cleaning app that scans your room and generates a timed task list), but the app itself makes zero LLM API calls. The scanning runs on Apple's Vision framework, entirely on-device. No tokens, no per-user inference costs, no latency, works offline, and my privacy label is basically empty.

Part of that was a deliberate bet: token costs for API-dependent apps feel unpredictable, and as a solo dev I didn't want my margins tied to a provider's pricing page. Every user costs me nothing to serve. I also hate the idea of sending images externally to an LLM.

But I'm aware I'm trading away capability. A cloud LLM could do far smarter things than my on-device pipeline can.

So, questions for anyone further down this road:

  1. If your app depends on external LLM APIs, how are you thinking about cost as you scale? Has pricing volatility bitten anyone yet?

  2. Anyone shipped with local/on-device models (Apple's Foundation Models, Core ML, quantised open models)? How was the quality gap in practice?

  3. Where's the line for you? Which features genuinely need a frontier model vs which are fine with something small and local?

And mostly: I'd love examples. What are the best shipped apps you've seen running LLMs locally or on-device? I want to study a few before I build my own local AI features, so drop names, even your own app. Especially your own app, honestly, self-promo welcome if it's actually running local inference.

Curious what people have found, especially other solo devs where inference cost comes straight out of your own pocket.

109 views

Add a comment

Replies

Best

Not running local, so can't speak to on-device but pricing volatility has bitten me for real. Cerebras announced deprecating a model I had in my main chain, on a deadline, same week they were also killing their free tier. Wasn't a "maybe someday" scare, it was "this breaks in 30 days."

Ended up walking away from that provider entirely rather than adding a card, and built a small internal gateway that reads model configs from a JSON file with deprecation dates baked in — so a provider killing a model doesn't take down a live feature, it just silently routes around it. Also added hard budget caps per provider so a traffic spike can't turn into a bill I didn't see coming.

Doesn't solve the margin question the way on-device does I'm still fully exposed to whatever providers charge but it turned "provider does something and my app breaks" into a non-event. If you do end up API-dependent for anything, that resilience layer is worth building before you need it, not after.

 oh wow the 30 day deadline thing is exactly the scenario that scared me off - I was worried about prices creeping up, hadn't even considered a provider just killing a model with a deadline. that's worse. The gateway idea is really smart tbh, deprecation dates baked into the config so it routes around dead models automatically. hard budget caps too - although a traffic spike turning into a surprise bill is one problem I'm fairly safe from, my users could triple overnight and the invoice would still be £0 (the number of users also helps 😄) saving all of this for if I ever do need an API for something - thanks for sharing it properly and not just saying "it depends"

 The deadline part really is the underrated risk price creep you can see coming and budget around; a provider just pulling a model out from under you with a countdown is a different kind of scramble. And fair, zero infra cost when it's £0 either way is a genuinely nice place to be no bill to protect against is its own kind of resilience.

Good luck with Briskly, and if you ever do need an API for something specific, the config-with-deprecation-dates pattern scales down fine too even a single provider benefits from not hardcoding a model name you'll have to hunt down and change everywhere the day it disappears.

Thank you! I will keep that in mind!
Hey Maria, this is the first Product Hunt post I read after creating my account, and the focus on on-device inference is exactly what sparked my interest. We built Talking Friday, a full speech-to-speech AI that runs entirely on the iPhone Neural Engine. No external LLM APIs at all. We ship three custom quantized models inside the app so the whole loop (listening, thinking, speaking) stays local. Privacy by design was a hard requirement for us: the voice never leaves the device, even for a second. The biggest constraint we keep running into is Apple’s memory limit for a single app. As conversations get longer the memory requirement climbs, and that has been our main engineering challenge so far. We’re planning a UX update next. If anyone wants to try the current version it’s at talkingfriday.com Curious how others are handling similar memory or size limits with local models!

 this sounds really cool! Also, three models running locally on the phone is pretty impressive. And yeah, the privacy thing is why I avoid voice input in the first place. I don't have the same problem to be fair. Mine's Vision framework doing image classification, so it's a quick burst with nothing building up over a conversation, just the accuracy isn't always the best. Your usecase sounds like a much harder version of it than anything I'm dealing with.

Bit outside my lane on the memory side, but I was wondering about the summarise-and-reset approach where you compress the conversation so far to clear the cache. Probably already on your radar. If it does mean a pause, that feels like a design problem as much as an engineering one - a voice assistant going quiet for a couple of seconds reads as broken unless something tells the user why. Might be less painful than trying to squeeze the memory down.

I think there’s an interesting middle ground for Briskly: keep the specialist vision pipeline, then give a small local language model the observations and a specific job—“I have 15 minutes, prioritise the kitchen, leave the noisy jobs until later.” That’s the kind of workflow I’d test before assuming it needs a frontier model.

I’m a founder of Lloyal, and we built it for developers shipping apps like these. You start with a working TypeScript app, with inference and an agent runtime built in. Lloyal handles model downloads, verification and packaging, so your users install an app without setting up an inference server or bringing API keys.

The model and application run in the same process. Agents can branch from context the model has already read, and you can compose specialist models for vision, retrieval and ranking with the reasoning model. The models become a part of your application and ship with it.

For a concrete example, our uses a 4B model on a laptop, alongside a reranker and vision projector, to investigate documents and produce cited reports. The quality gap still matters: I’d judge a small model against the actual decisions your app needs to make, including its mistakes, rather than its ability to answer anything.

Your photos are another relevant piece. Lloyal has a content-addressed media store: agents, the interface and saved results can reference the same stored image. For Briskly, that could keep a generated task connected to the exact room photo behind it, without uploading the photo to a model provider.

Our public release covers desktop, web and CLI; mobile is still in alpha.

, and I’d really value your feedback. What would you need to see before trusting a local LLM with Briskly’s task planning?