sysOne-devOS

Local "System One" decision model for Pi: the benchmark of Kev on the Radeon AI PRO R9700, the labelled request set, the fine-tuned and calibrated Kev-0.8B router, and the tooling that produced them. The router itself (checkpoint, CPU image, service, and a general guide to training your own) lives in the private repo Pcsf/kev-router. This repo is everything needed to retrain it from my own data.

• ISA.md: state of record. Every decision, number and method, newest first.
• kev-9b-r9700-results.md: the benchmark and fine-tune report.
• jev-local-decision-model-instructions.md: the original task spec (voice notes and sources).
• labels-review*-report.md (local only, not in git: they quote request text): my label reviews, applied by the bench/src/finalize-*.ts and build-trainset.ts scripts.
• bench/: TypeScript (bun) harness: extraction, labelling, synthetic data, training-set build, scoring, calibration, gate, latency, co-residency.
• docker/Dockerfile: the ROCm image (WITH_FLA=1 for the GPU, 0 for the CPU).
• kev_ext/: the park/unpark wrapper, and the first kev-router unit. The live unit now comes from Pcsf/kev-router (systemd/kev-router.service), installed by the kev-router module of omarchy-dev-bootstrap.

The dataset is not in git

Every file that contains my real requests is kept out of this repo and backed up as one zip in my vault, which syncs to Google Drive:

~/Documents/dDocs/2-resources/files/40-professional/42-ai-and-machine-learning/kev-router-dataset/kev-router-dataset-2026-09-27.zip
sha256 7b4536295dbe2428cc43c36bbab6dc218f7c683b9ec93d66af2d2ae6fcefe5f0

It holds 74 files (1.9 MB) with a MANIFEST.sha256 and a README.txt describing them. The important ones:

File	What it is
bench/data/labels.real.jsonl	780 reviewed real requests: session, text, previous assistant turn, label
bench/data/synthetic.final.jsonl, synthetic2.jsonl	synthetic rows, rounds 3 and 4
bench/data/trainset.meta.jsonl	the 1,159 rows chosen for training
kev-work/ft/train-full.jsonl, holdout-full.jsonl	the exact files Kev trained and was evaluated on
kev-work/ft/evalfull-*/	benchmark outputs; the predictions the calibration and gate were fitted on
bench/results/classify-*.json	classifier runs with per-request detail

The rest are earlier pools, pre-label rounds and review sheets, kept for provenance.

Restore

T=$(mktemp -d) && cd "$T" && unzip -q <path-to-zip>
sha256sum -c --quiet MANIFEST.sha256
cp -a sysOne-devOS/bench/data ~/Documents/10-repos/11-gitRepo/sysOne-devOS/bench/
cp -a sysOne-devOS/bench/results/. ~/Documents/10-repos/11-gitRepo/sysOne-devOS/bench/results/
mkdir -p ~/kev-work/ft && cp -a kev-work/ft/. ~/kev-work/ft/

Dependencies

• Kev checkout at ~/kev-work/kev, jaredpalmer/kev commit 5920c5f, bind-mounted into the container.
• Base model Qwen/Qwen3.5-0.8B-Base revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 and initial weights jaredpalmer/kev-0.8b revision 9a45d25eb2ab761841196625383fa1dff0e56c1e in ~/ai/models/managed/hf-hub, the Hugging Face hub layout kept by llama-model-manager. Fetch them with llama-model-add --format hf --repo <repo> --revision <commit> (plain curl, every file hash-checked).
• Images from docker/Dockerfile: kev-rocm:fla (--build-arg WITH_FLA=1) for training and GPU serving, kev-rocm:nofla for the CPU router. Rootless Docker: export DOCKER_HOST=unix:///run/user/1000/docker.sock.
• The R9700 is HIP device 0 (HIP_VISIBLE_DEVICES=0, gcnArchName == gfx1201).
• bun for bench/ (cd bench && bun install && bun test).

Retrain

1. Restore the dataset (above).

2. Rebuild the training and holdout files. This regenerates ~/kev-work/ft/train-full.jsonl and holdout-full.jsonl byte-identical to the originals (checked 2026-09-27):

   cd bench && bun src/build-trainset.ts
   

3. Train on the R9700. The arguments are those recorded in the checkpoint's training_config.json; about ten minutes, 4.7 GB of VRAM. kev.train refuses an existing --out, so pick a new name:

   docker run --rm --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --ipc=host \
     -e HIP_VISIBLE_DEVICES=0 -e HF_HUB_OFFLINE=1 \
     -v ~/kev-work/kev:/work/kev:ro -v ~/ai/models/managed/hf-hub:/root/.cache/huggingface/hub:ro -v ~/kev-work/ft:/ft \
     kev-rocm:fla python -m kev.train --base Qwen/Qwen3.5-0.8B-Base --base_revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 \
       --init_from jaredpalmer/kev-0.8b --data /ft/train-full.jsonl --out /ft/<new-run> \
       --device cuda --epochs 2 --lr 2e-5 --dtype bf16 --batch 1 --accum 8 --lora 16 --checkpointing 1 --max_state 384
   

   --base is required: without it kev.train falls back to Qwen/Qwen3-0.6B-Base, which is not cached, and fails offline. --device cuda is required because kev.train rejects --dtype bf16 before it auto-detects the GPU. A run that fails leaves an empty --out directory behind; remove it before retrying.

4. Evaluate on the holdout (same container flags): python -m kev.benchmark --run /ft/<new-run> --data /ft/holdout-full.jsonl --out /ft/evalfull-<new-run>, then bun src/score-predictions.ts new=$HOME/kev-work/ft/evalfull-<new-run>/predictions.jsonl. The reference run scored 0.813 accuracy, 0.683 macro-F1.

5. Calibrate: bun src/calibrate.ts <predictions.jsonl> ~/kev-work/ft/holdout-full.jsonl results/<name>.json, then write the mean fold temperature into a copy of the run with Kev's scripts/calibrate_checkpoint.py --run <copy> --rows <eval>/rows.json --temperature <T> --reason "...".

6. Check the gate: bun src/gate.ts <predictions.jsonl> ~/kev-work/ft/holdout-full.jsonl <T> results/<name>.json. The interim gate is p ≥ 0.85.

7. Serve: copy the calibrated run into Pcsf/kev-router's checkpoint/ (or point kev_ext/kev-router.service at it) and restart kev-router.

The method behind each step, and why it is done that way, is in ISA.md (Decisions and Learning). The same pipeline, generalised to anyone's own requests, is in Pcsf/kev-router's README.org.