Research Note · Updated June 5, 2026
Running Local Models at Home
The headline: On an 8GB Mac Mini, you can run five genuinely useful small models locally, but none of them is good at everything. Gemma 4 E2B is the overall winner at 82.2% weighted score. Bonsai 8B and LFM 2.5 are tied for second at ~79%. But the real finding is structural: the models' architectures are diverging so fast that "which model is best" depends on what you're doing, and the answer keeps changing.
Summary Results
Thirty prompts across eight categories, three temperatures, three repeats each = 90 scored items per model. Weighted score penalizes hallucination (3x) and broken tool calls (2x) harder than formatting mistakes. The table below shows each model's best temperature configuration:
| Rank | Model | Best Temp | Weighted% | Pass% | Strengths | Weaknesses |
|---|---|---|---|---|---|---|
| 1 | Gemma 4 E2B | 1.0 | 82.2% | 72.2% | Instruction following (97%), planning (100%), explanation (100%), context reasoning (100%) | Agent/tool-calling (40-53%) |
| 2 | Bonsai 8B | 0.3 | 79.4% | 71.1% | Agent tool-calling (60%), context reasoning (100%), fastest inference (2-bit quant) | 2-bit precision loss causes instruction drift on edge cases |
| 3 | LFM 2.5 | 0.3 | 78.9% | 70.0% | Hallucination resistance (83% at t0.3), multi-step tool chains, clean refusal behavior | Verbose reasoning (50-70% of output), sometimes hallucinates service names |
| 4 | Qwen 4B | 0.7/1.0 | 74.4% | 63.3% | Balanced across all categories, never spectacularly fails | Shallow on long prompts, no standout strength |
| 5 | Ministral 3B | 0.7 | 68.3% | 54.4% | Competitive on instruction following at t0.7 | 10+ points behind the next model in every category |
Key takeaway from the numbers: The top three are within 3.3 points of each other. That is not a decisive win — it is a cluster. Gemma leads on instruction-following and content generation. LFM leads on hallucination resistance. Bonsai leads on agent tool-calling and speed. The practical choice depends on which failure mode you can tolerate.
1. Why I Did This
The local inference space is moving fast. Six months ago, running an 8B model on consumer hardware meant watching a slideshow of barely-coherent text. Today, a $600 computer can serve useful output from several different architectures. But "useful" is not uniform — different models hallucinate in different ways, handle tools differently, and break on different kinds of prompts.
I wanted to understand the actual capability surface of these small open models without a vendor's marketing layer between me and what they produce. The only way to do that is to run them yourself, on your own hardware, with your own prompts.
2. The Models
The five accepted models, with their actual architectures:
| Model | Architecture | Active Params | Total Size | Source |
|---|---|---|---|---|
| Gemma 4 E2B (Google) | Dense (E2B = 2B effective) | 2B | ~2B | Local (LM Studio) |
| Bonsai 8B | 2-bit MLX quantized dense | ~8B (degraded precision) | 8B (2-bit) | Local (LM Studio) |
| LFM 2.5-8B-A1B (Liquid AI) | Mixture of Attention (MQA) | ~2.5B | 8B | Local (LM Studio) |
| Qwen 3.5 4B (Alibaba) | Dense | 4B | 4B | Local (LM Studio) |
| Ministral 3B (Mistral) | Dense | 3B | 3B | Local (LM Studio) |
Note on "8B": The class labels are misleading. Gemma 4 E2B is 2B with a clean training recipe. LFM 2.5 has 8B total but only ~2.5B active per forward pass. Bonsai is 8B quantized to 2 bits — fast, but lossy. Qwen is genuinely 4B. Ministral is 3B. Calling these "five 8B models" is wrong. They are five different architectures at five different sizes.
3. How I Tested
Thirty prompts across eight categories, each reflecting a task I actually perform — HOA board emails, code debugging, investment analysis, tool orchestration, technical writing. Each prompt has a rubric with pass/partial/fail criteria and a cost-of-failure weight (HIGH = 3, MED = 2, LOW = 1). A hallucinated financial planning recommendation weighs 3x more than a formatting issue in a summary.
Every model ran at three temperatures (0.3, 0.7, 1.0) with three repeats per setting. All local models used LM Studio with 8K max tokens and a 3-second pause between prompts to avoid OOM on the 8GB machine. Scoring was rubric-based via an independent scorer script — no LLM-as-judge contamination.
4. What I Found
Gemma 4 E2B — The All-Rounder (82.2%)
Gemma 4 E2B at temperature 1.0 was the strongest overall performer, scoring 82.2% weighted. It completed all 90 runs without error. Its outputs were the most concise and the most accurate on code, factual writing, and summarization. When it could not answer a question, it said so cleanly — no hallucinated service names, no fabricated entities. On instruction following, planning, and explanation prompts, Gemma scored 97-100%.
The caveat is that Gemma's token efficiency is partly an illusion. Google's AI Studio API reports completion_tokens that exclude hidden <thought> tags. On hard prompts, the real token count is approximately 8x higher than reported. Anyone modeling economics or context window usage needs to account for this.
Bonsai 8B — The Speed Runner (79.4%)
Bonsai 8B at temperature 0.3 scored 79.4% — functionally tied with LFM for second place. Its 2-bit quantization makes it the fastest model to load and run on the Mac Mini, and it scored 60% on agent tool-calling (the best of any model in the set). Its 100% on context-reasoning prompts tied Gemma.
The tradeoff for speed is precision. Bonsai occasionally interprets instructions oddly or skips subtasks, especially at higher temperatures. It is a workable default for straightforward tasks but unreliable when the prompt has multiple constraints.
LFM 2.5 — The Refuser (78.9%)
LFM 2.5 at temperature 0.3 scored 78.9%. Its standout dimension is hallucination resistance: 83% at low temperature, the best in the set. When it does not know the answer, it says so — no invented URLs, no fabricated services. This matters specifically for financial analysis and research tasks where a wrong answer is worse than no answer.
The tradeoff is verbosity. LFM emits its reasoning chain explicitly, and that reasoning is frequently 50-70% of total output tokens. On planning prompts, LFM sometimes hallucinates plausible-sounding service names (it once invented a fake email service called "Zervos" with fabricated DNS records). Gemma and Bonsai avoided this failure entirely.
LFM's "optimized for agentic use" claim from Liquid AI does not fully prove out in this benchmark — Bonsai leads AGENT scores, and Gemma ties LFM. Whether LFM's reasoning overhead is genuine improvement or learned deliberation that burns tokens is an open question.
Qwen 4B — The Median (74.4%)
Qwen 4B scored 74.4% at both 0.7 and 1.0 temperatures. It never failed spectacularly on any category, but it never excelled either. Its outputs tend toward the shallow end on long prompts — adequate summarization, routine code, basic instruction following — but with less depth than the top three.
Qwen is the "good enough" option. If you need to hand a model to someone who does not want to think about which model to use, Qwen is the safest default because it is the hardest to break. But it is also the hardest to love.
Ministral 3B — The Lastplace (68.3%)
Ministral 3B at temperature 0.7 scored 68.3%, 10+ points behind the next model. It is not broken — it completes runs, follows basic instructions, produces readable output — but it is outclassed by every other model in the set on every dimension. At temperature 1.0, its pass rate drops to 46.7%.
On an 8GB machine, there is no reason to use Ministral 3B when Qwen 4B and Gemma 2B load at similar speeds and score significantly higher across the board.
Supplementary Models
Three additional models were tested but did not qualify for the main ranking:
- NLD-3B (Nemotron Diffusion 3B) — run both locally (73.3%) and on Google Colab (71.7%). The Colab version scored 0.00 on hallucination resistance, the worst in the entire dataset. AR mode and self-speculation mode produced bit-identical outputs, suggesting the speculative decoding path is inactive in practice.
- ZAYA1-8B (Zyphra) — cloud API only. Agentic outputs comparable to LFM, but the cloud quota (100K tokens/day) expired before the full 30-prompt suite could complete. Partial score of 70.5% from 39 completed items. Zyphra's CCA attention mechanism shows promise but the token governor bug makes it impractical for agentic workflows.
- HRM-Text-1B — a 1B-parameter hierarchical reasoning model fine-tuned locally via QLoRA. Scored 61.7% after extensive iteration (see the separate writeup). Modest improvement over its base 58.3%, demonstrating the ceiling of 1B-parameter QLoRA on 8GB hardware.
5. The Unexpected Discoveries
Format Sensitivity Dominates Model Choice
Early runs used simplified tool definitions. Both LFM and Gemma reverted to their native tool-calling syntax instead of valid JSON, scoring near zero. After switching the prompt format to the OpenAI JSON schema, tool-calling scores jumped from ~14% to 64%+. The prompt format is not a minor variable — it is the dominant variable for agentic tasks.
Reasoning Overhead Is Unevenly Distributed
LFM exposes reasoning in visible tokens; Gemma's reasoning is embedded in weights. LFM's visible reasoning was frequently 50%+ of output, making it expensive on 8GB hardware. Gemma produces leaner outputs that are often better on the same dimensions. How much of LFM's reasoning is genuine improvement versus learned deliberation that burns tokens is an open question that matters for local deployment economics.
Google's Token Accounting Trap
Gemma 4's reported completion_tokens from the AI Studio API excludes hidden <thought> content. On hard prompts, the real token count is ~8x higher than reported. Anyone comparing costs across models needs to account for this, and most benchmark comparisons do not.
6. The Practical Outcome
I started this expecting one model to dominate. Instead I ended up with a split toolkit. Gemma for writing and code drafts. LFM when hallucination resistance matters. Bonsai when speed matters most. Qwen when I want the boring, reliable option.
The practical consequence is that I keep three models loaded in LM Studio and rotate them depending on the task. That is messy compared to a single default, but the outputs are meaningfully better than forcing one model to do everything.
For anyone building a local AI stack — whether for privacy, cost control, or experimentation — the honest takeaway is that local inference is now genuinely useful. The split-toolkit finding is structural, not temporary: small dense models optimized for writing, large MoE models for reasoning, quantized models for speed. The architectures are diverging, not converging. The local inference stack will look less like a single default and more like a toolbox, and that is fine.
Appendix: Methodology & Data Sources
Hardware: Apple Mac Mini M2, 8GB unified memory, macOS, LM Studio (version 3.10).
Scoring: Rubric-based independent scorer. 30 prompts, 3 temperatures, 3 repeats = 1,350 total scored items. Weighted formula: (2×P2 + P1) / (2×N) × 100. Categories: AGENT (tool calling), CODE, HALL (hallucination resistance), INST (instruction following), CTX (context reasoning), SUMM (summarization), PLAN, EXPLAIN.
Limitations:
- All models run on a single 8GB machine — results may differ on higher-end hardware with more memory bandwidth.
- LFM 2.5 at t0.7 tested with 4096 context (not 8192) — its t0.7 score may be deflated on long-context prompts.
- Models tested over late May—early June 2026. Weights may have been updated since.
- ZAYA, Nemotron, and HRM-1B results are supplementary — see their dedicated writeups for full methodology.
Raw data, code, and adapters: Full benchmark data, scoring scripts, HRM fine-tuning pipeline, and fine-tuned model adapters are available on GitHub (benchmark + code) and HuggingFace (fine-tuned adapters).
— the_red_deer