llm-bakeoff

• Dependencies
• Layout
• Usage
  • Before a run
  • Settings
• Adding things
• Reference result

Head-to-head benchmark for local GGUF models on this machine: quality, thinking tokens, wall time and decode speed, all measured under the same settings. Every item is graded deterministically (unit tests or a numeric answer within a tolerance), so no LLM judge is involved.

Dependencies

What	Version proven on	Used for
llama.cpp-vulkan	0.8.0-1, commit 6ebec9a (AUR build)	llama-server (Vulkan, draft-mtp)
Strata	ghcr.io/niko1221/strata:0.2.9-rocm-gfx120x	Flash-Next and Qwen3.6 reference runs
bun	1.4.0	runner, tests
python3	3.14.8	runs graded code; JSON parse in hf-get.sh
bubblewrap	0.13.1	sandbox for model-written code
coreutils	-	timeout, sha256sum
curl	-	model downloads, router checks
amdgpu sysfs	-	gpu_busy_percent for the contention guard

Hardware the configs in bench.json assume:

• Vulkan0 = Radeon AI PRO R9700 32 GB eGPU (DRM card1)
• Vulkan1 = Radeon 890M iGPU (DRM card2, also drives the desktop)
• Model files under ~/ai/models/managed/ (llama-model-manager layout)

Layout

Path	Content
bench.json	server port, contention guard, server env, model configs
src/suite.ts	the 18 items and their hidden tests / answers
src/grade.ts	code extraction, bwrap sandbox run, numeric check
src/bench.ts	runs the suite against the server, appends JSONL rows
src/summarize.ts	per-model totals and a per-item table
src/serve.ts	starts llama-server for one model config
src/wait-health.ts	waits for the server's /health
src/guard.ts	contention guard shared by both benches
src/agent-suite.ts	tool, multi-turn and long-context cases, stub tools
src/agent-run.ts	streaming client and the case runners
src/agent-bench.ts	runs the agent cases, appends JSONL rows
src/agent-summarize.ts	agent totals and a per-sample table
scripts/hf-get.sh	resumable HF download, sha256 checked against the LFS oid
test/	config test, agent test and grader selftest
results/	reference runs
runs/	scratch output, git-ignored

Usage

bun run test                          # config, agent and grader tests, no GPU needed
bun run serve tc > runs/srv-tc.log 2>&1 &
bun run wait
bun run bench tc runs/tc.jsonl 3      # label, output, samples per item [, item regex]
bun run summarize runs/tc.jsonl tc
bun run agent tc-agent runs/tc-agent.jsonl 3   # tool and chat k samples, long-context 1
bun run agent:summarize runs/tc-agent.jsonl tc-agent

To test a server this repo did not start, set BENCH_URL to its address and BENCH_KEY to its API key if it has one. Reasoning tokens are recorded as null when the server has no /tokenize.

To compare against a reference run, concatenate the files and use a different label per run: cat results/*.jsonl runs/new.jsonl > runs/all.jsonl, then bun run summarize runs/all.jsonl tc new.

bench resumes: it skips label/item/sample rows already in the output, so rerunning after a dropped request fills only the gap. bun run serve <model> --print shows the exact command without starting anything.

Stop the server by its PID, never by a name pattern.

Before a run

The bench server (port 8090) must have the GPUs to itself. Leave the router services (llama-server.service on 8080, llama-server-igpu.service on

1. running, because Vigil raises CRIT when they stop, and unload their

models with llama-model-unload instead. Before each sample the guard waits until both cards are below their busy threshold and neither router has a model loaded. A sample taken after maxWaitS without that is recorded with env.ok=false and counted as noisy by summarize.

Settings

Every request: temperature 1.0, top-p 0.95, top-k 20, min-p 0, max_tokens 32768, thinking on, one request at a time. Reasoning tokens are counted by tokenizing reasoning_content on the same server.

Adding things

• A model: add an entry to models in bench.json (the model path and its args; ~/ is expanded), then a test in test/config.test.ts pinning the args.
• An item: add it to src/suite.ts, and add a correct and a plausible but wrong solution to test/selftest.ts. The selftest must show the good one passing and the wrong one failing, or the grader proves nothing.

Reference result

Quality suite, 18 items x 3 samples unless noted, files under results/:

File	Model and engine	pass	tokens mean	wall mean	decode
2026-10-22-thinkingcap	ThinkingCap Q4_K_M, llama.cpp Oct	50/54	3790	74 s	57.1 t/s
2026-11-01-tc-080	same, llama.cpp 0.8.0, 1 sample	15/18	6105	122 s	55.9 t/s
2026-11-01-strata-fn	Flash-Next IQ3_XXS, Strata 0.2.9	53/54	10031	129 s	84.3 t/s
2026-11-01-qwen36	Qwen3.6-35B UD-Q4_K_M, llama.cpp 0.8.0	50/54	12553	70 s	181.0 t/s

Agent test, 6 tool and 3 multi-turn cases x 3 samples, 3 long-context cases x 1:

File	Model and engine	tool	chat	long	tool wall	chat wall	long first answer, 10K/40K/80K tokens
2026-11-01-tc-agent	ThinkingCap	18/18	9/9	3/3	3.3 s	8.4 s	10.3 / 46.3 / 109.1 s
2026-11-01-strata-agent	Flash-Next, Strata	18/18	9/9	3/3	3.2 s	7.2 s	5.0 / 17.9 / 33.6 s

Engine speed, results/2026-11-02-engine-speed.jsonl: 200 tokens after a prompt of 10K tokens (3 samples) and 41K tokens (1 sample), t/s. Strata does not run ThinkingCap. Strata's Qwen3.6 is the UD-Q4_K_M file under the UD-Q4_K_XL name it asks for.

Model	Engine	prefill 10K	prefill 41K	decode 10K	decode 41K	VRAM
ThinkingCap	llama.cpp	994	876	60.5	48.7	26.6 GiB
Flash-Next	llama.cpp	479	444	36.0	30.8	30.0 GiB
Flash-Next	Strata	1956	2255	88.1	87.6	31.4 GiB
Qwen3.6-35B	llama.cpp	3048	2503	166.2	161.7	22.1 GiB
Qwen3.6-35B	Strata	5464	3071	94.9	35.7	29.1 GiB