Edge AI: Running LLMs on Smart Home Devices (2026 Guide)
Yes, you can run a local LLM as your Home Assistant conversation agent today — but only on hardware with enough RAM, only with small models, only with fewer than 25 exposed entities, and with the honest caveat that Home Assistant calls this control feature experimental and no vendor publishes tokens-per-second figures for the hardware most homeowners own. The path runs through the official Home Assistant Ollama integration, which requires an external Ollama server rather than a model on the Home Assistant box itself. Realistic hardware starts around 8 GB of RAM for a 4b-class model. Cloud assistants still win for long conversations and complex requests.
This guide is based on official Home Assistant and Ollama documentation and published model listings. We did not benchmark these models on hardware ourselves. Every figure below comes from vendor documentation, and where a performance number is not vendor-published, we say so explicitly instead of guessing.
How the Ollama integration actually works
The Ollama integration “adds a conversation agent in Home Assistant powered by a local Ollama server.” It requires a separate Ollama install, works only with Tools-capable models for control, and Home Assistant labels that control feature experimental.
The key word is external: the integration does not run a model inside Home Assistant. Ollama runs as its own server — available for macOS, Linux, and Windows — which can live on the same machine as Home Assistant or on another box on your network. The Ollama FAQ notes the server must be configured to be accessible over the network.
Three constraints matter before you start:
- Only Tools-capable models can control Home Assistant. Not every small model supports tool calling; check the model listing before you download.
- Home control is experimental. Home Assistant says so directly on the integration page.
- The context window defaults differ. Home Assistant defaults its context window to 8k tokens, while the Ollama server defaults to 2k. Larger context supports larger homes; smaller values lower Ollama’s RAM usage.
If you want the full picture of the voice hardware around this, our local voice assistant Home Assistant build log walks the pipeline end to end.
What hardware is realistic
Our working threshold is 8 GB of RAM for a 4b-class model, and that is our inference from the download sizes. Home Assistant Green is a capable hub but not an LLM server. No vendor publishes tokens-per-second for the hardware here, so treat any speed claim you see elsewhere as unverified.
The Raspberry Pi 5 comes in 1 GB, 2 GB, 4 GB, 8 GB, and 16 GB RAM variants. The 16 GB model is listed at $305, and Raspberry Pi commits that it “will remain in production until at least January 2036.” We did not extract prices for the other variants.
Home Assistant Green, at a recommended MSRP of $199 USD / €179 incl. tax, ships with 4 GB LPDDR4X RAM. That is enough for the Home Assistant hub itself. Home Assistant does not present it as an LLM host, so it is the hub, not the model.
Two distinctions homeowners consistently miss:
- Download size is not runtime RAM. The file sizes on Ollama’s model pages are quantized downloads. At runtime, RAM usage is higher because of the context window and KV cache. A 2.5 GB download does not mean 2.5 GB of free RAM is enough.
- Speed numbers don’t exist for the hardware you probably own. No vendor publishes tokens-per-second for the Pi 5, Intel N100, or Jetson in the sources we reviewed. We won’t estimate.
Also note the contrast with our Raspberry Pi AI HAT local object detection guide: that accelerator handles object detection, not LLMs — a different job entirely.
| Device | RAM | Can run | Cannot run |
|---|---|---|---|
| Home Assistant Green ($199/€179 MSRP) | 4 GB LPDDR4X | Home Assistant itself, Speech-to-Phrase | An LLM (not presented as an LLM host) |
| Raspberry Pi 5 8 GB | 8 GB | Small models; a 4b-class model at the edge of comfort (our inference) | Larger models; no published tokens/second |
| Raspberry Pi 5 16 GB ($305) | 16 GB | 4b–8b; 14b tight (our inference from download sizes) | 30b-class models (our inference) |
| Intel NUC / desktop, 16 GB+ | 16 GB+ | 8b–14b (our inference); faster Whisper STT (under 1 s per HA docs) | 30b-class at high context (our inference) |
| Jetson-class | Not researched | No sourced figures | No sourced tokens/second |
Which models make sense
Small models are the only realistic local option, and Home Assistant warns they are more likely to make mistakes than larger ones. A 4b-class model is the practical sweet spot for an 8 GB machine; the smallest tiers suit only very constrained setups where you accept the accuracy trade-off.
Home Assistant’s own warning bears repeating: “smaller models are more likely to make mistakes than larger models.” Locally, you don’t get to skip that trade — you manage it. Here are the download sizes Ollama lists for Qwen3 and Gemma 3:
| Model | Download size | Notes |
|---|---|---|
| qwen3:0.6b | 523 MB | 40K context; smallest tier, most mistake-prone |
| qwen3:1.7b | 1.4 GB | 40K context |
| qwen3:4b | 2.5 GB | 256K context; practical sweet spot for 8 GB machines |
| qwen3:8b | 5.2 GB | 40K context |
| qwen3:14b | 9.3 GB | 40K context |
| qwen3:30b | 19 GB | 256K context; desktop-class only |
| gemma3:270m | 292 MB | 32K context |
| gemma3:1b | 806 MB | 32K context |
| gemma3:4b | 3.4 GB | 128K context, text+image |
| gemma3:12b | 8.2 GB | 128K context |
| gemma3:27b | 17 GB | 128K context |
These are download sizes, not runtime RAM — at runtime, usage is higher because of the context window and KV cache. The Qwen3 listing tags its models for tools, which home control requires. The Gemma 3 listing we read does not show a tools tag, so check a model’s tags before you use it for control. Start with a 4b-class model on 8 GB hardware; drop to 0.6b–1.7b only if your hardware demands it and you accept more mistakes.
The voice pipeline around the LLM
The LLM is one link in a chain — speech-to-text, then the LLM, then text-to-speech — and the slowest link sets your experience. The Assist voice control overview describes the full pipeline. Speech-to-Phrase answers in under a second on Home Assistant Green or a Pi 4 but handles a limited command set; Whisper takes about 8 seconds on a Pi 4. A fast LLM behind slow STT still feels sluggish.
Home Assistant’s local assistant documentation lays out the two STT paths clearly:
- Speech-to-Phrase is closed-ended and fast — “under one second” on a Home Assistant Green or Raspberry Pi 4 — but it supports only a subset of Assist commands. Shopping lists, timers, and broadcasts are not usable out of the box.
- Whisper is open-ended but slow on small hardware: roughly 8 seconds per command on a Pi 4, and under one second on an Intel NUC. Home Assistant calls Whisper “only a great option” with powerful hardware and if you plan to pair it with an LLM-based agent — which is exactly the setup this guide describes.
For text-to-speech, Piper runs locally and is optimized for the Raspberry Pi 4.
On the satellite side, ESPHome voice assistant components require Home Assistant 2023.5 or later, and the documentation is blunt: audio and voice components consume significant RAM and CPU, “crashes are likely,” and Bluetooth/BLE components are known to cause issues when combined with Voice Assistant. Our Home Assistant Voice Preview Edition review covers the purpose-built alternative, and once your pipeline works, our Home Assistant automations beginner guide shows what to do with it. The Voice Preview Edition page lists the dedicated hardware itself.
Exposing entities and controlling your home
You must explicitly expose entities to Assist before the LLM can control them, and Home Assistant recommends exposing fewer than 25. More entities mean a larger context and a higher cost per request, so a lean exposure list is both a safety measure and a performance one.
The exposing devices guide explains why exposure is opt-in: it keeps sensitive devices like locks and garage doors from being controlled inadvertently. The best practices page adds the performance angle — exposing more entities “will incur a higher cost per request, due to the larger context size” with an LLM agent.
This creates the central tension of a local setup: you want a small model because RAM is scarce, but Home Assistant itself warns smaller models are more likely to make mistakes. Your defense is a short entity list. Start with lights and switches — not every sensor, not your lock. Fewer than 25 entities, per Home Assistant’s recommendation.
What still needs the cloud
Home Assistant itself frames the trade-off. It warns that smaller models are more likely to make mistakes than larger ones, and it notes that more exposed entities raise the cost per LLM request because the context grows. Those two points are the sourced limits. A small local model on 8 GB of RAM is a reasonable fit for simple commands, and it is the weaker fit for long, multi-step requests.
The Voice Preview Edition page describes a lower-risk option: an LLM as “a fallback for commands that Home Assistant does not understand natively,” rather than a full replacement for the built-in assistant. Start there if you are unsure. Our judgment, not a sourced claim: if your daily use is “turn off the living room lights,” local is viable, and if you chain requests or ask follow-up questions, a cloud model will likely serve you better.
FAQ
Can Home Assistant Green run a local LLM?
No. Home Assistant Green is an easy-start hub with 4 GB LPDDR4X RAM, and Home Assistant does not present it as an LLM host. The Ollama integration needs an external Ollama server on a separate machine, and the 8 GB figure for a 4b-class model is our inference from the download sizes, not a vendor requirement.
Is local LLM control of my home safe?
Home Assistant labels home control through the Ollama integration experimental, recommends exposing fewer than 25 entities, and warns that smaller models are more likely to make mistakes. Exposure is explicitly opt-in, which keeps locks and garage doors out of reach unless you add them. Treat it as an experiment: start with lights and switches only.
How fast is local voice with an LLM?
Speech-to-Phrase responds in under one second on Home Assistant Green or a Raspberry Pi 4, but with a limited command set. Whisper takes about 8 seconds on a Pi 4. No vendor publishes tokens-per-second for local LLMs on Pi 5, N100, or Jetson hardware, so the LLM’s contribution to response time is an honest, documented gap.
The Bottom Line
Running a local LLM as your Home Assistant conversation agent is real and worth trying if you have 8 GB or more of RAM, a 4b-class Tools-capable model, fewer than 25 exposed entities, and realistic expectations about experimental control. If what you actually want is fast, reliable voice commands today, the Speech-to-Phrase pipelines in our local voice assistant build log and Voice Preview Edition review are the mature path. The LLM layer is the frontier — promising, but not yet the default.
How This Guide Was Built
This guide draws on official Home Assistant documentation (the Ollama integration, local assistant, exposure, and best practices pages), Ollama’s model listings, ESPHome’s voice assistant docs, and the Raspberry Pi 5 product page — all read on 2026-10-15. No hands-on benchmarking was performed. Gaps we named rather than estimated: no vendor-published tokens-per-second for Pi 5, N100, or Jetson; no measured end-to-end Home Assistant plus LLM latency; and the Hailo and AI HAT+ 2 vendor pages were unavailable at read time.
← Back to guides