kev-router

A local request router for Pi coding agents. A fine-tuned, calibrated Kev-0.8B reads each request and answers one question: is the user asking for code, for a plan, or for a discussion? The Pi packages pi-lifeos and pi-devos use the answer to hold coding turns to test-first (pi-devos also requires an ISA.md first).

This repository holds the checkpoint, the image and service to run it on a CPU, and the tools and steps to train your own from your own requests. The checkpoint was trained on one person's requests. Expect it to fit someone else's traffic worse than these numbers show, and retrain it.

Results

Measured on 384 real requests from 83 sessions that were never used for training. Accuracy is over all three classes.

Model	Accuracy	Macro-F1	Coding recall	ECE
Always "discussion"	0.703	-	0	-
Stock jaredpalmer/kev-0.8b	0.555	0.457	0.814	-
Stock jaredpalmer/kev-9b	0.815	0.626	0.619	0.078
This checkpoint, uncalibrated	0.813	0.683	0.680	0.132
This checkpoint, calibrated (T = 2.85)	0.813	0.683	0.680	0.042

The calibrated ECE is out of fold (see Calibration below). Temperature never changes the top answer, so accuracy is identical.

Nothing here reaches 90% on every request. What makes the router usable is a confidence gate: act on Kev's answer only when its top probability is at least 0.85, and send everything else to a fallback. Out of fold, that acts on 52% of requests at 95.0% accuracy (95% interval 91.1–97.3%), and it stayed above 93% on 20 different random splits. The threshold 0.85 was chosen after the 0.8 and 0.9 results were known, so treat it as a well-supported estimate, not a pre-registered result.

Serving on a CPU (Ryzen AI 9 HX 470, 24 threads, bf16): p50 773 ms, p95 929 ms on real requests, 3.7 GB of RAM, no VRAM. The Pi packages send the request when the turn starts and only need the answer at the first file write, so the agent never waits for it. On a Radeon AI PRO R9700 the same model answers in under 80 ms, but a router that is always resident then competes with local LLMs for VRAM.

What is here

checkpoint/

the LoRA adapter (r = 16) and pointer head on top of Qwen/Qwen3.5-0.8B-Base, with the temperature written into head.pt. 63 MB. Kev loads the base model by repository id and revision from the Hugging Face hub cache, downloading it at first start if it is missing.

docker/Dockerfile

the image the results were measured with. Build it without flash-linear-attention for CPU serving.

systemd/kev-router.service

a user unit that keeps the server running.

tools/

TypeScript (bun) tools for the steps below: extracting requests from agent logs, splitting by session, scoring, calibration and the gate threshold.

Dependencies

Everything below is what the results were produced with.

• Kev source: jaredpalmer/kev at commit 5920c5f, checked out locally and mounted into the container. Kev needs Python 3.12–3.13 and torch < 2.9.
• Base model Qwen/Qwen3.5-0.8B-Base revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68, initial weights jaredpalmer/kev-0.8b revision 9a45d25e.
• Image base rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.8.0 (digest pinned in the Dockerfile): torch 2.8.0+rocm7.2.4, transformers 5.17.0, peft 0.21.0. It runs fine without a GPU. A CPU-only torch image should also work but was not tested.
• flash-linear-attention 0.5.2 only for GPU serving or training (WITH_FLA=1). It must be absent for CPU serving: its Triton kernels fail on the first request when no GPU driver is present.
• Rootless Docker, and bun 1.x for the tools.
• For training and fast evaluation: an AMD GPU with ROCm (the R9700, gfx1201, was used) or any GPU Kev supports. Training peaked at 4.7 GB of VRAM.
• Hugging Face's Python client stalled on the machine used here. If downloads hang, fetch the files with plain resumable curl into the hub cache layout. On that machine llama-model-add --format hf from llama-model-manager does this, into ~/ai/models/managed/hf-hub.

Run the router

Build the CPU image and start the server:

docker build -t kev-router:cpu docker/          # WITH_FLA defaults to 0
docker run -d --name kev-router -e KEV_DTYPE=bf16 \
  -p 127.0.0.1:8009:8009 \
  -v <kev-checkout>:/work/kev:ro \
  -v ~/.cache/huggingface/hub:/root/.cache/huggingface/hub \
  -v "$PWD/checkpoint":/ck:ro \
  kev-router:cpu python -m kev.serve --run /ck --host 0.0.0.0 --port 8009
curl -s 127.0.0.1:8009/v1/models    # device cpu, dtype bfloat16, temperature 2.8497

Once the base model is cached, add -e HF_HUB_OFFLINE=1 so a restart never reaches the network, and the hub cache can be mounted read-only (:ro). systemd/kev-router.service does both and takes the hub cache directory from KEV_HF_HUB, which defaults to ~/ai/models/managed/hf-hub. Do not pass /dev/kfd or /dev/dri; without them Kev cannot allocate VRAM.

To keep it running, edit the two paths at the top of systemd/kev-router.service, then:

cp systemd/kev-router.service ~/.config/systemd/user/
systemctl --user daemon-reload && systemctl --user enable --now kev-router

Then switch the router on in the Pi package, as its README describes (router.enabled in ~/.pi/agent/pi-lifeos.json or pi-devos.json). Under pi-docker, launch with --local so the container can reach 127.0.0.1:8009.

The question the router asks is fixed. Changing a single character of it invalidates the temperature and the threshold, because both were fitted to this exact wording:

What kind of turn is the user asking the coding assistant for?
  coding:     Create or change code, scripts, RTL, tests or program configuration in this turn
  planning:   Produce a plan, a design, or an opinion on an approach before anything is implemented
  discussion: Answer a question, explain, report status, or run non-code actions such as git commits, notes or hardware checks

The state sent with it is the user's request (at most 1,200 characters) and the end of the previous assistant message (at most 400). Kev was trained on states up to 384 tokens and rejects longer ones.

Train your own

1. Collect requests

cd tools && bun install
bun src/extract-pi.ts ~/.pi/agent/sessions > candidates-pi.jsonl
bun src/extract-claude.ts ~/.claude/projects > candidates-claude.jsonl

Each line is one request with its session id and the last 400 characters of the assistant message before it. Drop harness noise (injected notices, pasted command output with no request in it) and exact duplicates. This work used 780 labelled real requests. A first fine-tune on 88 of them reached only 0.750 on held-out requests.

2. Label them

One label per request: coding, planning or discussion. The class definitions are the question above. These boundary rules settled the cases that kept coming back, and writing yours down early saves relabelling later:

1. Classify the current request. The previous assistant message is context, not the thing being classified.
2. A failure report on something just built in that session is coding, because a repair is implied. An observation the assistant asked for, or a hypothesis to investigate, is discussion.
3. An approval ("yes", "do it") takes the class of the action it approves.
4. Running tests, git operations, notes, moving files and bench checks are discussion. Editing a script, config or skill file is coding.
5. Planning means asking for an approach or a recommendation. A request that both plans and authorizes implementation is coding.
6. The deliverable decides diagnosis against planning: explain, find the cause or inspect is discussion; recommend what to check first, how to investigate, or which fix to take is planning.
7. When unsure, leave it out rather than guess. Held-back rows hurt less than wrong ones.

Label everything yourself or pre-label with a model and review every row. Save the result as JSONL with session, text, context and label.

3. Split by session

bun src/split.ts labels.jsonl out/ [synthetic.jsonl]

Sessions go to one side or the other by a fixed hash, never both, because turns of one conversation are close to copies of each other and would inflate the held-out score. out/ receives train.jsonl, holdout.jsonl and holdout.meta.jsonl (the session of each held-out row, used by calibration). Synthetic rows only ever go into training. The held-out rows are evaluated once per decision; do not tune anything on them except through the cross-fits below.

4. Optional: synthetic training rows

Real planning requests are rare (4% here), so the classifier sees few examples. 773 synthetic rows were generated with a local LLM, seeded only from requests in the training sessions so nothing held out could leak through paraphrase. What went wrong and how it was fixed:

• The generator put "plan", "propose", "sketch", "outline" or "architecture" in 69% of synthetic planning rows, against 19% of real ones, and the model learned the words instead of the intent. The fix: generate planning requests with those words banned, add coding and discussion rows that use them, and cap the explicit-wording planning rows at the real share.
• Near-duplicate families (token Jaccard ≥ 0.5) were capped at two members, and rows too close to any real request (Jaccard ≥ 0.8) were dropped.
• Every synthetic row was screened against the boundary rules, and the flagged ones (about 10%) were reviewed by hand.

5. Train

On a GPU, from the Kev checkout:

python -m kev.train --base Qwen/Qwen3.5-0.8B-Base --base_revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 \
  --init_from jaredpalmer/kev-0.8b --data out/train.jsonl --out runs/mine \
  --device cuda --epochs 2 --lr 2e-5 --dtype bf16 --batch 1 --accum 8 --lora 16 --checkpointing 1 --max_state 384

--base is required: without it kev.train falls back to Qwen/Qwen3-0.6B-Base, a different model. --device cuda is required because kev.train rejects --dtype bf16 before it auto-detects the GPU. A run that fails leaves an empty --out directory behind; remove it before retrying.

1,152 rows took 288 optimizer steps and about ten minutes on the R9700. kev.train refuses to overwrite an existing --out directory.

6. Evaluate

python -m kev.benchmark --run runs/mine --data out/holdout.jsonl --out runs/mine-holdout
bun src/score-predictions.ts mine=runs/mine-holdout/predictions.jsonl

Report accuracy next to the always-majority baseline, macro-F1 and per-class recall. With 70% of requests being discussion, accuracy alone hides a model that never predicts the rare classes. Decide what counts as useful before you look; the criterion used here was a clear margin over the baseline and coding recall at least as good as stock Kev-9B.

7. Calibrate

A fresh fine-tune is overconfident. Here, 10% of all answers were wrong at p ≥ 0.9, which makes a confidence gate useless. Every training request was used for training, so the temperature has to come from the held-out rows without scoring it on the rows it was fitted on:

bun src/calibrate.ts runs/mine-holdout/predictions.jsonl out/holdout.meta.jsonl calibration.json

This splits the held-out sessions in two, fits the temperature on one half, scores it on the other, and swaps. Report the out-of-fold ECE. If the two fold temperatures agree, write their mean into a copy of the checkpoint:

cp -a runs/mine runs/mine-cal
python scripts/calibrate_checkpoint.py --run runs/mine-cal --rows runs/mine-holdout/rows.json \
  --temperature <T> --reason "mean of the cross-fit fold temperatures; out-of-fold ECE <x>"

Kev's own fitting mode refuses rows it cannot place as held out, which includes any custom dataset, so the value is computed here and passed with --temperature.

8. Choose the gate

bun src/gate.ts runs/mine-holdout/predictions.jsonl out/holdout.meta.jsonl <T> gate.json

This picks the smallest threshold reaching 90% on one half and scores it on the other. Expect it to land just under 90%: here it gave 89.7%, because a threshold that barely reaches the target on one half barely misses it on the other. The two halves also chose 0.76 and 0.66, so a few hundred rows cannot place a threshold closer than about ±0.05. Pick a threshold with margin above that edge, and confirm it later on requests collected after training.

9. Serve and wire in

Serve the calibrated copy as in "Run the router", then switch the router on in pi-lifeos or pi-devos. Both packages have a live check (bun scripts/smoke-router.ts) that runs a coding and a discussion prompt through pi-docker and reports the route Kev chose for each.

Limits

• Trained and evaluated on English requests from one person's Pi and Claude Code sessions. Portuguese was left out on purpose.
• Planning is thin: 17 held-out planning requests, recall 0.47. Treat planning answers with care.
• The held-out set served both the evaluation and the calibration. The nested splits keep each number honest for what it measures, but a new set of requests is the real test.
• Weights: Apache-2.0, as Kev and the Qwen3.5 base are. See checkpoint/LICENSE.