The New Pi AI HAT and What It Can Really Do

TL;DR

• The AI HAT+ 2 is a $130 Hailo-10H board with 40 TOPS INT4 and 8GB of dedicated RAM, built for local LLMs and vision-language models on a Pi 5.
• Official guidance caps useful language models around 1B to 7B parameters. Launch models were mostly 1B and 1.5B. Do not expect ChatGPT-class answers.
• Independent tests show the Pi 5 CPU often decodes tokens faster than the NPU. The HAT wins on prefill speed, power draw, and keeping the CPU free.
• Vision work (YOLO, pose, segmentation) is the proven path. Language and voice agents are promising if you accept small models and private data.
• Buy it for privacy, offline operation, or CPU offload. Skip it if you only need vision, or if a 16GB Pi 5 with llama.cpp already fits your model.

What you are actually buying

The newest board in Raspberry Pi's AI line is not a faster version of the old one. It is a different product with a different job.

The original AI HAT+ shipped with Hailo-8L or Hailo-8 silicon at 13 or 26 TOPS. Those boards accelerate vision models. Object detection, pose estimation, segmentation. No language models.

The AI HAT+ 2, launched 15 January 2026, swaps in a Hailo-10H and adds 8GB of LPDDR4X memory on the HAT itself. Raspberry Pi quotes 40 TOPS at INT4. That memory is the real news. The accelerator can hold a small language model without eating the Pi's own RAM.

A few specs worth memorizing:

• Hailo-10H, 40 TOPS INT4, roughly 3W maximum draw on the NPU
• 8GB dedicated LPDDR4X on the board
• PCIe Gen 3 link to a Raspberry Pi 5 (Pi 5 only; not Pi 4)
• Launch price $130; a later July 2026 product brief lists $200, so check current pricing before you order
• Vision performance roughly on par with the 26-TOPS AI HAT+

That last point matters. If your project is camera-only, the cheaper AI HAT+ or the $70 AI Camera still do the job. The + 2 earns its price when you want language models, vision-language models, or a clean split where the NPU works while the Pi CPU runs your application stack.

Jeff Geerling's launch-day testing makes the trade concrete. Same 8GB Pi 5, same models, CPU versus NPU. Vision ran fine. Language was more complicated. We will get to those numbers shortly.

The size ceiling: how big a model can you run

Memory is destiny on edge hardware. The AI HAT+ 2 has 8GB on the module. After runtime overhead and context, you are looking at weights that fit comfortably under that line.

Raspberry Pi's own documentation puts the practical ceiling near 6 billion parameters for LLMs and VLMs on this board. Their launch announcement was blunter: edge language models on the HAT typically run at 1B to 7B parameters, while cloud models sit in the 500B to 2T range. Different class of machine.

Launch models were few.

Larger compiled models were promised after launch. Even so, the design point is clear. These are narrow, task-shaped models. Summarize a note, classify intent, rewrite a paragraph: fine. Hold a large world knowledge base? No. And they will invent facts when pressed outside their training distribution.

Tom's Hardware hit this wall in review. Ask qwen2:1.5b who Tom's Hardware is and both the HAT and the Pi CPU invented a wrong answer. A generated Python image-display script from the HAT looked tidy and still failed to open the image.

None of that makes the board useless. It makes the model choice part of the engineering. Treat a 1.5B model like a junior teammate with a short memory and a good checklist. Give it constrained tasks. Keep humans in the loop for anything factual or high stakes.

Quick sizing math for quantized weights: many 4-bit downloads land near 0.6GB per billion parameters. A 1.5B model often weighs 1-2GB on disk. Expect 4-5GB for a 7B model at 4-bit, plus context. Plan headroom. A 4.7GB model does not live happily in 5GB of free space.

The benchmark reality: NPU versus Pi CPU

Here is the part marketing pages skip.

Raw token generation on a small model is memory-bandwidth bound. Pi 5 LPDDR4X and the HAT's on-module memory sit in the same league, but the Hailo-10H is capped near 3W while the Pi SoC can push closer to 10W. When Geerling ran every available Hailo model against the Pi CPU on identical 8GB hardware, the CPU won most decode comparisons. Only Qwen2.5-Coder 1.5B came close.

Academic work agrees. Edge-LLM-Bench, presented at a MobiSys 2026 workshop, calls the platform the first sub-5W discrete accelerator that can run billion-parameter models on a commodity SBC. One finding is sobering. The Hailo-10H does not improve raw token throughput over the host CPU for small language models.

So why does the board exist?

Three reasons show up in both vendor data and independent measurement.

First, prefill is different from decode. Prefill processes your prompt. Decode emits tokens one by one. Hailo reports time-to-first-token of 320ms on the NPU versus 2039ms on the Pi CPU for Qwen2.5 1.5B at 4-bit. If your app takes a long document, a camera frame, or a voice transcript and produces a short answer, prefill-heavy profiles favor the HAT.

Second, power and consistency. An arXiv cross-platform study measured Qwen 2.5 1.5B at 4-bit on the Hailo-10H sustaining about 6.9 tokens per second under 2W, with near-zero variance across runs. A laptop RTX 4050 hit 131.7 tokens per second at 34.1W. Different products for different jobs. The NPU's value is stable, low-power, always-on inference.

Third, offload. When the NPU handles inference, the Pi CPU stays responsive for cameras, Home Assistant, networking, or your own code. Geerling notes this as the honest win. Mixed vision-plus-language mode was still rough at launch, with segmentation faults in his testing, but the architectural intent is sound.

Hailo's own guidance lines up with the data. Their blog positions the + 2 for compute-heavy encoder stages and short responses, not endless chat. Voice-to-action pipelines (Whisper-class speech to text, then a small LLM for intent) and vision-language captioning are the patterns they expect to win.

Tools that make the HAT earn its keep

Hardware is half the project. Software decides whether you ship something.

hailo-ollama plus Open WebUI. The default language stack from Raspberry Pi's announcement. hailo-ollama exposes an OpenAI-style API on the HAT. Open WebUI gives you a browser chat UI in front of it. Tom's Hardware pulled models with a plain curl against port 8000. Good for private chat over local notes and quick experiments.

rpicam-apps and Picamera2. The native vision path. Raspberry Pi OS detects the accelerator and routes post-processing to the NPU. Geerling's Camera Module 3 test tagged a desk scene at around 30fps, roughly ten times the CPU. This is the most mature workflow on the board.

Hailo Dataflow Compiler. Needed when stock models do not fit your domain. Retrain YOLO-style detectors on your own images. Compile LoRA adapters onto a base language model so a 1.5B model learns your vocabulary without a full fine-tune. Raspberry Pi documents LoRA support as the intended fix for the small-knowledge problem.

Ollama and llama.cpp on the CPU. Do not treat these as competitors to the HAT. Treat them as the other tool in the drawer. A 16GB Pi 5 running a compressed Qwen3 30B A3B in about 10GB of RAM generated a working drag-and-drop TODO app for Geerling, a task the tiny HAT models handled poorly. Different budget, different tool.

Frigate. Local network video recording with object detection, integrated with Home Assistant. Proven on the earlier AI HAT+ with pinned Hailo driver packages. If you want a 24/7 camera brain that never phones home, Frigate already works with this family of boards. Confirm 10H support in your Frigate version before you buy for that purpose alone.

Whisper-class speech models. Voice agents need a speech-to-text stage before any LLM sees intent. Hailo's materials describe this pipeline as a strong fit because transcription is encoder-heavy. Community discussion around the launch pointed at English models in the few-gigabyte range as realistic on the 8GB module.

Stack rule of thumb: pick the smallest model that clears your task quality bar, keep context windows modest, and move prefill to the NPU. Leave decode-heavy chatty workloads on the CPU when speed matters more than watts. Smaller wins here.

Six projects worth building

Concrete ideas, ordered from safest to most ambitious.

1. Private document Q&A. Chunk meeting notes or a PDF set, embed them locally, and answer questions with a 1.5B instruct model through Open WebUI. Nothing leaves your network. Quality improves fast when answers stay close to the provided text.

1. Camera event logger with captions. Run YOLO on the NPU, then let a vision-language model write a one-line description when something interesting appears. Prefill-heavy, short output. The profile the hardware likes.

1. Voice switch for home automation. Record a command, transcribe it with a Whisper-class model, extract intent with a small LLM, trigger Home Assistant. Bounded vocabulary. Fast responses. No cloud speech bill.

1. Offline code assistant for airgapped machines. Qwen2.5-Coder 1.5B on the HAT is the one launch-era model Geerling found nearly competitive with the CPU. Short completions and explanations. Ideal when the machine never touches the internet.

1. Frigate NVR with local alerts. Object detection stays on the HAT. An optional language step turns event metadata into a daily digest. Read it from your phone.

1. LoRA-tuned domain classifier. Collect a few hundred labeled examples from your workshop, factory line, or shop floor. Compile a LoRA adapter. Now a tiny model speaks your jargon instead of the open internet's.

A minimal LLM bring-up looks like this on Raspberry Pi OS Trixie:

sudo apt update
sudo apt install hailo-h10-all
hailortcli fw-control identify

Once the driver answers, pull a model through the hailo-ollama API the same way Tom's Hardware did in review:

curl --silent http://localhost:8000/api/pull \
  -H 'Content-Type: application/json' \
  -d '{ "model": "qwen2:1.5b", "stream": true }'

Point Open WebUI at that backend and you have a private chat UI with no cloud account attached. No account. No egress.

Every one of these projects shares a boring failure mode: hallucinated specifics. Constrain outputs. Cite sources from your own data. Keep a human approval step for anything that triggers a physical action.

Toy or tool: a decision framework

The fair question is not whether the AI HAT+ 2 is real. It is real hardware with measured behavior. The question is whether it fits your constraint set.

Buy it when data must stay on premises, when the Pi CPU is already busy, or when your workload is prefill-heavy with short answers. Vision-language pipelines and voice agents on small vocabularies are also solid matches. Some people will want it purely as a development kit before shipping Hailo-10H inside a larger product. Fit beats hype.

Skip it when you only need object detection. The older AI HAT+ or AI Camera costs less and does that job. Skip it if your model does not fit in about 6B parameters at usable quantization. A 16GB Pi 5 running llama.cpp will likely beat it on raw generation speed for models that big.

And treat early-adopter tax as real. Software caught up slowly at launch. Geerling's mixed-mode attempts crashed. Tom's Hardware saw HailoRT readiness errors. Raspberry Pi and Hailo ship updates on a steady cadence, but expect to pin driver versions and read release notes.

Is it a toy? For general chat, yes, compared with a cloud model. Private, offline, power-aware inference at the edge is a different story. There the board is a capable tool with a short model catalog and honest physics. People who ship on Pi have always matched the tool to the task instead of forcing one board to do everything.

If that sounds like your bench, the AI HAT+ 2 is worth a slot. Start with vision. Add a small language model when the data privacy argument is strong enough to justify the quirks.

Sources

• Introducing the Raspberry Pi AI HAT+ 2 (Raspberry Pi)
• Raspberry Pi AI software documentation
• Jeff Geerling: Raspberry Pi's new AI HAT adds 8GB of RAM for local LLMs
• Tom's Hardware: Raspberry Pi AI HAT+ 2 Review
• Edge-LLM-Bench (ACM MobiSys workshop, EMDL '26)
• LLM Inference at the Edge (arXiv preprint)
• Hailo: When and why you'll need the AI HAT+ 2