A while back I wrote about swapping InfluxDB for TimescaleDB and called it paying the LLM tax. The short version was that once an LLM is in the loop, the boring infrastructure decisions stop being about my convenience and start being about what the model can actually work with. I’ve been living in that tradeoff long enough now to turn it into something more useful than a gripe.
Every AI feature I build gets squeezed by the same three forces. Quality, cost, and latency. You can usually have two. The whole job is picking which two, on purpose, for this specific feature.
Local Or Hosted Is Not A Religious Question
I run a lot of AI locally on my own GPUs with Ollama, and I have opinions about privacy. But when I’m wearing the product hat instead of the homelab hat, local versus hosted is a tradeoff, not an identity. Four signals decide it, and Nova is the easy case because all four point the same direction. High constant volume, a voice assistant that can’t afford a cloud round trip, house data I’d rather keep in the house, and a job small enough that a local model handles it fine.
The harder cases are when the signals disagree.
| The signal | Lean local | Lean hosted |
|---|---|---|
| Query volumehow often it runs | High and constant. The GPU is already paid for. | Occasional. Not worth dedicated hardware. |
| Latencyhow fast it must feel | A cloud round trip would wreck the experience. | A short wait is fine. |
| Data sensitivitywhere it can go | It genuinely should not leave the house. | Not sensitive, or already lives elsewhere. |
| Task difficultyhow smart it must be | A small model is good enough for the job. | You need the smartest model you can get. |
Four signals. When they point different directions, that is the decision.
Trying to force a tiny model onto a task it can’t handle to save a few cents is the most expensive kind of cheap. The reverse is just as common and less obvious, which brings me to the leg of the triangle everyone skips.
What A Query Actually Costs
Cost gets ignored until the bill shows up, because per-query pricing looks like nothing. Fractions of a cent. Then you multiply by volume and it stops being a rounding error.
Cruxwire is a great example. It pulls 58 feeds, scores every item for relevance, and runs semantic deduplication across the whole set, and it does that every two hours between 6am and 10pm. So it isn’t one batch I can forget about, it’s 2,800+ items a day, and each one is a model call. At hosted API prices that’s about $65 a month even on Claude Haiku for output nobody is waiting on and nobody but me will ever read. On Nova it’s roughly fifty cents a month in marginal electricity, on a GPU that was already spinning for other reasons.
The interesting part isn’t that local came out cheaper. It’s that volume made the call, not quality. Scoring an RSS item is not a hard task. A small local model does it well enough that paying frontier prices would be spending real money to improve an answer nobody grades. Flip the volume and the answer flips with it. If Cruxwire ran once a week I never would have thought about the GPU. The fifty cents is also only a rout because the hardware was already there for Frigate and everything else Nova runs. If I had bought the box for this one job, hosted would have won for years before the capex paid itself back.
Latency Is A Feature
The thing PMs underweight is that latency isn’t a technical footnote, it’s part of the product. A voice assistant that answers in three seconds feels broken even if the answer is perfect. A daily garden briefing that takes thirty seconds to generate is completely fine because nobody is standing there waiting.
So the latency budget depends entirely on the interaction. Nova needs an answer before I’ve walked out of the room. Ceres can think as long as it wants overnight. Same underlying tech, wildly different constraints, and if you don’t set the budget up front you find out the hard way when the demo feels sluggish.
The Hardware Is Part Of The Product Too
I felt this most going from the Coral to the Hailo for Frigate object detection. It wasn’t about running more camera streams. I’d run out of patience for false positives, and the detection model that actually fixed that was bigger than the Coral could hold. That’s not a hobby upgrade, it’s a capability decision. The model I wanted set the floor for the hardware I needed, and pixels processed per detection turned out to be the number that mattered.
Model choice, hardware, cost per query, and latency aren’t separate conversations. They’re one conversation, and the PM who treats them as one makes better calls than the one who picks a model and then acts surprised by the bill.
The Framework I Actually Use
Before I build, I answer four things.
- How good does the answer have to be
- How fast does it have to come back
- How often will this run
- What does one run cost at that volume
Most of the time those four answers pick the model and the place to run it for me. When they conflict, that conflict is the actual product decision, and it’s worth making out loud instead of by accident. The failure mode is choosing a model first and meeting the other three later, in a bill or in a demo that feels slow.
