I have wanted an Alexa that doesn’t phone home for years. Every commercial voice assistant is a microphone in your house wired to someone else’s computer, and the whole point of a homelab is that my stuff runs on my stuff. So I built one. It’s called Nova, it lives on my LAN, and nothing it hears has to leave the property.
The interesting part wasn’t the tool itself. It was making a small local model trustworthy enough to run a house, which is a product problem, not a coding problem. This post is about the decisions, what broke, what the eval numbers actually say, and what I’d do if I did it again.
The Stack
The short version: a web app, a FastAPI backend, three AI services on my local GPU server, Raspberry Pi satellites listening in the shop and the house, and a routing layer that decides which of eleven skills a request belongs to before any model sees it.
The brains are faster-whisper for STT, Ollama serving qwen3:8b for the LLM, and Kokoro for TTS. An 8B model is the constraint the whole product is shaped around. It’s a capable home-assistant brain and an unreliable everything-else, so every interesting decision below is about keeping it inside the box where it’s good.
Routing Without a Router Model
A lot of local-assistant projects throw every request at the LLM with every tool attached and hope. With an 8B model, hope does not deliver. The more tools qwen3 sees, the dumber it gets about all of them.
Nova routes deterministically before the model is involved. Eleven skills, each with a keyword list. The matcher does whole-word matching so “turn on” doesn’t fire inside “return online.” Nothing matches? The turn is plain chat with zero tools, which keeps the small model fast and honest. A follow-up detector catches pronouns like “it” and “again” so “turn them back off” inherits the previous turn’s skill, and the keyword lists pull live vocabulary from the environment (Home Assistant device names, manual titles, camera names) so “pull up the driveway” opens a camera feed without the word “camera” appearing anywhere.
I did consider the fancier options. I actually built an LLM classifier as a routing fallback, measured it, and deleted it. More on that below. The core objection stands on its own: using the unreliable 8B model to route around the unreliable 8B model was the problem I was trying to solve, not a solution to it.
Once a skill routes, the model gets only that skill’s tools and runs a bounded agent loop: stream, collect tool calls, execute, feed results back, up to five rounds, then force an answer. Scoping the toolset is what makes a small model usable at all.
Tools That Can’t Lie (Much)
Small models have failure modes you have to design against directly. My favorite: qwen3 will announce “let me search for that” and then just not call the tool. A regex catches these announced-but-didn’t-call replies and injects one internal nudge the user never sees. Every turn also carries an honesty instruction: do not claim an action you didn’t take. It mostly works. Mostly.
Home Assistant control has a real safety layer. My instance has around 2,000 entities, and early on the model happily toggled the wrong ones. Now the assistant only sees entities I’ve explicitly labeled nova in Home Assistant, injected with exact IDs, and the backend refuses any tool call touching an entity that isn’t on the list. Curating is just labeling things, no config file, no restart. The principle generalizes: don’t ask a small model to be careful, remove its ability to be dangerous.
Same instinct shows up in smaller places. Math goes through a deterministic calculator tool instead of letting the model compute, because a model that guesses at arithmetic is worse than no answer.
When the Small Model Taps Out
qwen3 faceplants on long tool chains and deep reasoning, so Nova has a hybrid brain behind one toggle: auto-escalate. An upfront regex catches requests that obviously want a big model (research, long-form writing, “ask Claude”) and routes straight to a frontier model, streaming live.
The reactive path is sneakier. qwen3 answers first, but the backend buffers its text while streaming only the tool activity. If the reply comes back empty or matches a punt detector (“would you like me to,” “consult the manufacturer,” and a dozen friends), the buffered answer is discarded and a cloud model retries with the same tools. You never see the bad answer get replaced, just a good one.
The alternative was running a 14B model locally full-time. I went with 8B plus opt-in escalation instead, and the eval results below back that call: the local model’s misses are exactly the hard multi-step cases that escalation exists to cover, so paying the bigger model’s latency tax on every turn buys almost nothing.
Escalation is a privacy tradeoff, so the product is loud about when it’s being made. Every reply is badged with the model that produced it, and when a turn leaves the LAN, the UI says so.
Does It Actually Work?
I built an eval harness, so here is what it says. The corpus is 55 hand-authored cases, each a real utterance paired with what the agent should do: which skill, which tool with what arguments, and whether the reply stays grounded. It runs against the real routed loop with a fake executor so nothing in the house actually toggles. Running qwen3:8b:
- Tool selection: 51/54 (94%) on the cases that assert a tool call. The four misses are all multi-step hard cases, which is exactly the boundary auto-escalate exists to cover.
- Skill routing: 55/55. Before anyone is impressed, that’s a claim about my keyword rules covering my own test set, not model magic. The 94% is the real number.
- Think-tag leaks into spoken output: 0/55. The scrubber holds.
Latency, measured warm against the live services: STT around 0.3s for a short clip, first LLM token around 0.2s, TTS around 0.1s for the first sentence. The pipeline streams sentence by sentence, so the first spoken word lands under a second after you stop talking.
Two honesty caveats. This is a hand-built corpus, not production traffic, and the harness doesn’t score grounding quality yet. It’s a report card, not a hard gate, since regressions in model behavior are a different problem than regressions in code. It also scores per model, so I can compare qwen3:8b against larger candidates before swapping brains. As for how often escalation actually fires in daily use, the honest answer is rarely, by design.
What Broke, and What I Changed
Four things I shipped, watched fail, and fixed.
The router. Version one handed qwen3:8b all 26 tools. It free-associated. “Read me today’s tasks” came back “the capital of France is Paris.” So I grouped tools into intent-scoped skills with a hybrid router: keywords first, an LLM classifier as fallback. Then the classifier itself mis-routed “capital of France” to web search and cost a second GPU call to do it. So I deleted it. Routing is now pure deterministic keyword matching, and it’s both faster and more accurate than either version that came before it.
Home Assistant. Fuzzy-searching 2,000 entities toggled the fireplace light instead of the kitchen island. That’s what drove the labeled-allowlist design above. The model lost the ability to guess, which is different from being asked to guess carefully.
Manuals. Embeddings alone weren’t enough. nomic-embed-text scored a Rikon band saw query against my Grizzly table saw chunks at 0.6 to 0.7 cosine similarity, a wrong-device match out-scoring correct ones, so no threshold could separate them. I added a deterministic brand and model token gate on top of the vector search. A miss now falls back to asking which model I mean, or searching the web, instead of confidently citing the wrong manual. Pure vector search was the rejected option here, and it earned the rejection.
Honesty. qwen3 once told me the lights were off without calling a tool. That incident produced the always-on honesty guardrail, the pronoun follow-up inheritance, and a unit test containing the exact sequence that regressed, so it can never silently break again.
The pattern across all four: when the model was unreliable, the fix was never a better prompt. It was moving the decision somewhere deterministic.
Grounding: Manuals and Cameras
Two features exist to give the model something true to stand on. I upload PDFs, the backend chunks and embeds them into a Qdrant vector store, and a search_manuals tool returns passages tagged with title and page. Asking for a torque spec gets a cited answer instead of a hallucinated one.
Cameras work the same way. “Pull up the front porch” opens a live feed, and the browser never touches the camera service directly. The backend proxies it and refuses any camera ID not in the live catalog. Same rule everywhere: one origin, and the model can only reach curated, real data.
One Conversation, Every Surface
The server owns the session. The client sends only the new message plus a conversation ID, and the backend rebuilds the full thread. That one decision means a conversation started by voice in the shop shows up in the web client, and I can continue it from the couch with full context.
History older than seven days is pruned because a voice-assistant transcript is not an heirloom. Memory is separate: durable facts in three tiers (today, this month, long-term) injected into every turn. History is what was said; memory is what should be remembered.
Timers make the same point about coherence. “Set a timer for 10 minutes” is easy; having it fire when no browser is open is the real feature. State lives server-side, the web client and the in-room satellite both poll a fired feed, and for my phone the backend hands the intent to Home Assistant over MQTT, which owns delivery as native push. I considered Web Push and went with MQTT to Home Assistant instead, because HA already owns notification policy in this house and I didn’t want a second delivery system to babysit. Nova emits the event; something durable delivers it.
Even the UI is built for observability, not just chat. The constellation view visualizes model flow in real time, every reply is tagged with the tool calls and model that produced it, and the settings tab keeps a request log so I can see exactly what was called and why.
What It Cost
The git history says 160 commits so far, roughly 16k lines. Hardware was stuff I already owned: the 2080 Ti handles STT and TTS, the 4070 runs the LLM, and each satellite is a Raspberry Pi and a speakerphone, about $150 per room.
If I rebuilt it tomorrow, I wouldn’t cut anything, but I would change the order. The eval harness would come first instead of last, because every architecture decision above got easier to make once I could measure it. The observability pieces (the constellation view and the request log) would come second, since debugging a small model without them is guesswork. The link dashboard would still make the trip too. It started as a standalone tool I open every day, and migrating it into Nova put the assistant on a surface that makes it sticky.
Shipping It
Nova runs as one Docker container behind LAN-only TLS, and HTTPS isn’t cosmetic. Browsers refuse mic access outside a secure context, so no TLS means no voice on phones. A push to main builds the container, health-checks it, then deploys the satellite code and restarts the service. Secrets never leave the box. A unit suite (no network, no GPU, no model) gates every push and covers the pure seams where bugs live: the router and its heuristics, the escalation detectors, the token scrubber, session rehydration, the payload sanitizer.
Where It Goes From Here
It answers, controls the house, sets timers from the shop, reads my manuals back to me, and when it hits something hard it quietly asks a bigger brain and tells me it did. That’s most of the Alexa feature set I actually used, plus a few new ones.
Next up: catching the turns where the local model is confident but wrong. The punt detector only catches it being humble and wrong, which is the easy case. Confident-and-wrong is the real frontier, and it’s the same problem everyone shipping AI products is stuck on. I just get to fight it in my own house.
