We benchmarked 8 small local LLMs for RAG honesty on a 2-vCPU server

Short answer: on a 2-vCPU server with no GPU, the small local models we tested split into two groups — fast ones that fabricate, and honest ones that are slow. The single change that helped every model was not swapping the model at all: it was fixing retrieval so the right passage actually reaches the model. granite3.3:2b was the only model whose quality held up across two machines and survived a fabrication test that broke the others, but its p95 latency (37,666 ms) sits above a comfortable budget on 2 vCPUs.

We run Smartbotify on a budget VPS with locally-run models and no per-token API bill, so "which small model can we trust" is a question we had to answer with measurements rather than vibes. Here is what we found.

The test

• Hardware: x86 2-vCPU GitHub-hosted runner — half of our 4-vCPU target box, so every number here is a conservative lower bound. No GPU.
• Task: answer 20 questions against an ingested corpus. 10 are answerable from the corpus; 10 are deliberately unanswerable and the model should decline.
• Metrics: correct (answerable questions answered right), honesty (unanswerable questions correctly declined, out of 10), unsupported (answers containing claims not in the corpus), and p95 response time.
• Gate: honesty ≥ 8/10 and p95 ≤ 60,000 ms at this tier.

Results

Model	Size	p95 (ms)	Honesty	Correct	Tenant-name leak
qwen3:1.7b	1.7B	14,870	10/10	6–7/10	4/8 leaks
granite4:3b	3B	30,480–33,075	7–8/10	6/10	5/8 leaks
granite3.3:2b	2B	37,666	9/10	7/10	0/8 clean
phi4-mini	3.8B	62,292	8/10	6/10	0/8 clean
granite4.1:3b	3B	stable	10/10	see below	0/4 genuine
qwen3:4b-instruct (primary)	4B	119,916 solo	high	see below	0/8 clean
qwen3:0.6b	0.6B	fast	7/10	—	genuine fabrications
gemma3:1b	1B	fast	4/10	—	genuine fabrications

qwen3:0.6b and gemma3:1b were rejected on honesty alone — both produced genuine fabrications, not just borderline phrasing. phi4-mini has acceptable quality but is hopeless on CPU at 62 seconds p95.

Finding 1: the fastest models fabricate the most

qwen3:1.7b answers in 15 seconds and declines every unanswerable question — a perfect 10/10 honesty score. It is also the least trustworthy model in the set, because the honesty metric only measures declining. A model that declines everything scores 10/10. Grading the answerable side too is what separates an accurate model from an evasive one.

Finding 2: models restate the customer's company name as biography

The sharpest failure was subtle enough that our first test fixture hid it entirely.

When the system prompt contains a line like "You are the assistant for Acme Logistics," small models lift that company name out of the instructions and restate it as a fact — "[Person] currently works at Acme Logistics" — even when the corpus never says so. The leak rate tracks how plausible the company name looks:

Company name in prompt	granite4:3b	qwen3:1.7b
Eval Fixture Co eval-fixture-1785316808 (obviously fake)	0/4	0/4
Acme Support	1/4	1/4
A normal-sounding company name	4/4	3/4

Our early evaluations used an absurd synthetic company name, so no model ever echoed it and every model looked clean. Real customers have normal names. This failure is live in production for any model that shows it, and it does not show up unless you test with a realistic name.

Neither temperature: 0 nor an explicit prompt clause ("the company name is not a fact about anyone in the context") fixed it. Only better retrieval helped, and only partially.

Finding 3: passing every automated check is not the same as being honest

granite4.1:3b passed every automated gate — 10/10 honesty across three local runs plus CI, 0/4 genuine name leaks, stable latency. Then we fact-checked its answers to the 10 answerable questions by hand, which the automated metric does not do. It got the subject's current employer wrong in 3 out of 3 runs, and in 3 out of 3 runs it invented a professional certification by lifting the credential from a colleague's testimonial quote and attributing it to the wrong person. Not recommended as a primary or fallback model.

The automated honesty score is good for screening out the worst candidates. It is not sufficient on its own to choose the model you ship.

Finding 4: the biggest lever was retrieval, not the model

Five of the ten answerable questions failed for every model. The diagnostic showed why: the correct passage existed and scored 0.588–0.648 — comfortably above the 0.5 relevance threshold — but it ranked 5th to 10th, outside the top 3 the system passed to the model. The model never saw the answer.

Raising the retrieval cutoff from the top 3 passages to the top 10 (with a larger context window) fixed two of the three worst questions for every model tested, with no regression on the controls. One example, before and after:

Question: "Where is [the person] based?" Top-3 retrieval: "The provided context does not mention [the person]'s location." Top-10 retrieval: "[The person] is based in Bengaluru, Karnataka, India."

That is a one-line configuration change worth more than any model swap.

Finding 5: concurrency is survivable, but cold starts 503

A 5-way concurrent burst on a laptop-class machine completed 5/5 with no timeouts; the slowest concurrent request took 2.47× the solo baseline. One reliable failure mode: the first request after a model swap returns a 503 on cold start. Production needs a warm-up ping after any model change.

What we actually run

• Primary: qwen3:4b-instruct for answer quality. Its solo p95 at 2 vCPUs (119,916 ms) sits at the timeout ceiling, so this is the open risk on the smallest tier.
• Pre-approved fallback: qwen3:1.7b — 10/10 honesty, p95 ~15 seconds. If the primary is too slow under real load, this is a one-line switch. Its name-leak weakness is mitigated by the retrieval fix and a conservative prompt.
• Retrieval: top-10 passages, not top-3. This did more for answer accuracy than every model comparison combined.

If you are choosing a small local model for RAG

1. Test with a realistic tenant/company name, not an obviously synthetic one. The dominant fabrication mode is invisible otherwise.
2. Grade the answerable questions, not just the declines. A model that refuses everything scores perfectly on a decline-only honesty metric.
3. Fact-check a sample of answers by hand. Automated scoring missed a model that got a key fact wrong in 100% of runs.
4. Fix retrieval before you swap models. If the right passage is not in the context window, no model can answer.
5. Never compare latency across CPU architectures. In our runs one model beat another on an Apple laptop and lost to it decisively on x86. Quality metrics transfer between machines; latency does not.

How Smartbotify applies this

Smartbotify is a self-hosted RAG chatbot platform: you add your PDFs and web pages, it handles chunking, embeddings and retrieval, and it answers only from your content — declining when the answer is not there. The model runs locally, so there are no per-message API fees. The decisions above are why it retrieves widely and prefers to say "I don't know." See how it works or the FAQ.