llama-model-manager

Versioned management tools for a local llama.cpp multi-model server.

Models are downloaded directly from Hugging Face into one persistent store. The server reads an explicit runtime registry instead of discovering cache entries, which prevents duplicate models in /v1/models.

Layout

Repository                          Versioned scripts and service definition
~/ai/bin/                           Stable symlinks to repository scripts
~/.local/bin/                       Commands available on PATH
~/ai/models/managed/                GGUF and multimodal projector files
~/ai/models/managed/hf-hub/         PyTorch/safetensors repositories, Hugging Face hub layout
~/.config/llama/presets.ini         Live model registry
~/.cache/llama-router-empty/        Empty cache used to disable discovery
~/.pi/agent/                        Pi configuration, extensions, agents, prompts

The repository also tracks the configuration itself, under config/ and pi/. GGUF files, backup files, credentials, and per-machine state are not stored in Git.

Installation

./install.sh --dry-run
./install.sh

The installer is idempotent. It backs up conflicting destinations before creating links, preserves an existing presets.ini, and copies the Qwen request-profile extension into Pi.

Configuration in Git

config/ and pi/ hold the tuned configuration: both preset registries, both service units, and Pi's models.json, settings.json, agents, prompts and extensions. Tuning a model for a specific GPU placement is real work, and losing it to an unversioned file is the failure this guards against.

llama-config-sync moves those files between the host and the repository:

llama-config-sync status        # what is linked, which copies differ
llama-config-sync pull          # host -> repository, then commit
llama-config-sync push          # install the links, restore the copies

Two registries and two service units are symlinked into this repository:

host path	repository path	lane
~/.config/llama/presets.ini	config/presets.ini	eGPU, llama-server.service
~/.config/llama/presets-igpu.ini	config/presets-igpu.ini	iGPU, llama-server-igpu.service
~/.config/systemd/user/llama-server.service	systemd/llama-server.service	eGPU
~/.config/systemd/user/llama-server-igpu.service	systemd/llama-server-igpu.service	iGPU

Each name points at its own file; the two lanes never share one. A symlink is one file under two names, so llama-model-add and llama-model-remove write the repository directly, nothing needs capturing, and the two copies cannot disagree. llama_install_temp_presets resolves the link before mv, which would otherwise replace it with a regular file on the first model added.

Everything under ~/.pi/agent is copied instead: Pi runs in a container that bind-mounts ~/.pi/agent but not this repository, so a symlink there names a path that does not exist inside the container and the file silently never loads. Run pull after changing any of those, then commit.

Both directions refuse rather than destroy. push will not replace a regular file whose contents differ from the repository's, and will not overwrite a copied file whose host version is newer; it names what is in the way. pull captures a host file that is not yet a link, which is the migration path onto links. --force overrides either refusal.

auth.json, models-store.json, sessions/ and lifeos-state/ are never tracked or touched.

The resulting link chain is:

~/.local/bin/llama-model-add
  -> ~/ai/bin/llama-model-add
  -> <repository>/bin/llama-model-add

~/ai/bin remains available for unrelated local tools.

Add a model

llama-model-add \
  --repo deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --file ornith-1.0-35b-Q4_K_M.gguf \
  --ctx-size 512000 \
  --restart

For multimodal models:

llama-model-add \
  --repo unsloth/Qwen3.6-27B-MTP-GGUF \
  --file Qwen3.6-27B-Q4_K_M.gguf \
  --mmproj-file auto \
  --restart

Import an existing GGUF below ~/ai/models without downloading it again:

llama-model-add \
  --repo ORG/REPO \
  --no-download \
  --file model-Q4_K_M.gguf \
  --restart

The import moves the file into managed storage. Use --dry-run to inspect paths and settings first.

Add a safetensors repository

PyTorch models (for example Kev's base model and adapter) are loaded by transformers by repository id and revision, not by llama-server. They are kept in the Hugging Face hub layout under ~/ai/models/managed/hf-hub, pinned to a full commit hash:

llama-model-add --format hf --repo Qwen/Qwen3.5-0.8B-Base \
  --revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68

Every file listed by the Hugging Face tree API at that commit is downloaded with resumable curl and checked before it is kept: sha256 for LFS files, the git blob sha1 for the rest. A mismatch deletes the partial download and leaves no repository behind. presets.ini is never touched.

Move a repository that is already in ~/.cache/huggingface/hub (or $HF_HUB_CACHE) instead of downloading it again; LFS blobs are re-hashed on the way in:

llama-model-add --format hf --repo ORG/REPO --revision COMMIT --no-download

llama-model-list shows these repositories in a separate table, llama-model-check reports broken snapshot links, and removal deletes the directory (there is no registry entry to drop):

llama-model-remove --format hf ORG/REPO --delete-files

A container loads them by mounting the directory as its hub cache, read-only, with downloads disabled:

docker run ... -e HF_HUB_OFFLINE=1 \
  -v ~/ai/models/managed/hf-hub:/root/.cache/huggingface/hub:ro ...

On the host, set HF_HUB_CACHE=~/ai/models/managed/hf-hub and HF_HUB_OFFLINE=1.

Model residency and switching

The server runs one child process per model and keeps it alive for sleep-idle-seconds after its last request, holding that model's device memory the whole time. A large model therefore starves whatever loads next.

A child decides where its tensors live when it loads, and never revisits that choice. A model that loads while the cards are full puts its weights in host memory and keeps generating from there even after the cards empty. Measured on one 27B, same server, same preset:

	CPU	eGPU busy	eGPU VRAM	speed
loaded while another model held the cards	55%	9%	332 MiB	3.5 t/s
loaded onto free cards	2%	97%	25806 MiB	28.2 t/s

CPU high with the GPU near idle is the signature of this failure.

sleep-idle-seconds is per-model, set like any other preset key. Give a large model a short window so it releases the cards soon after use:

llama-model-add --repo ORG/REPO --no-download --file MODEL.gguf \
  --arg 'sleep-idle-seconds=60' --restart

Measured: a 87 GiB model at 60 released both cards 65 s after its last request, with no command run, and the next model then loaded onto free cards at full speed. It cannot help when models are switched within that window, because placement is already committed by then.

It must not go on the service command line, for the same reason as --device: arguments there beat every preset.

llama-model-unload

For switching sooner than the timeout, or for diagnosing a slow model:

llama-model-unload --list          # what is loaded, and how much it holds
llama-model-unload                 # unload everything
llama-model-unload MODEL-ID        # unload one
llama-model-unload --keep MODEL-ID # unload everything else

Children are matched by resolved executable plus --alias, never by a command-line pattern, so the command cannot match the shell running it; the router has no --alias and is excluded by construction. SIGTERM is ignored by these children, so SIGKILL is used, and the command waits for the processes to leave before returning.

Unload the starved model too, not only the one holding the memory. Freeing memory around a running child does not move it back onto the GPU. With no arguments this command does that by default.

Memory still "used" after unloading a split model

After a model that spans the iGPU is unloaded, btop can still show tens of GiB of system memory in use with no model loaded. That memory is not held by any llama-server process: the kernel's TTM page pool keeps the freed GTT pages for reuse, up to half of RAM, and restarting the service does not return them. Measured after unloading the Flash-Next split: 36.2 GiB in the pool, all on the 890M, with no llama-server holding that device open.

The pool is reclaimable, so the kernel takes it back under memory pressure, but MemAvailable does not count it. To release it on demand:

llama-model-unload --release-pool   # unload, then release; one sudo prompt
sudo sh -c 'sync; echo 2 > /proc/sys/vm/drop_caches'   # the same, by hand

The flag releases even when nothing was left to unload, and prints the pool size before and after. Without a terminal, sudo asks through $SUDO_ASKPASS, defaulting to the seahorse dialog. It is opt-in because it needs root, and because level 2 also drops the dentry and inode caches system-wide.

Level 2 runs the shrinkers, including TTM's, and leaves the page cache (and so the cached model files) alone. sudo cat /sys/kernel/debug/ttm/page_pool shows the pool size in pages. A permanent cap needs ttm.page_pool_size=<pages> on the kernel command line: the limit is read once at module load, so writing the sysfs parameter at runtime has no effect.

Multi-GPU placement

Placement is per-model, set with --arg:

llama-model-add --repo ORG/REPO --no-download --file MODEL.gguf \
  --arg 'LLAMA_ARG_DEVICE=Vulkan0,Vulkan1' \
  --arg 'LLAMA_ARG_SPLIT_MODE=layer' \
  --arg 'LLAMA_ARG_TENSOR_SPLIT=1,0' \
  --arg 'LLAMA_ARG_OVERRIDE_TENSOR=big_tensor=CPU,blk\.(1[0-9]|[2-9][0-9])\.ffn_(gate|up|down)_exps=Vulkan1' \
  --restart

llama-server and llama-bench use different separators for the same two options, and llama-server accepts the wrong form without applying it:

option	llama-server	llama-bench
device list	Vulkan0,Vulkan1	Vulkan0/Vulkan1
-ot rules	a=CPU,b=Vulkan1	a=CPU;b=Vulkan1

A benchmark string pasted into a preset therefore produces no error and no split: every tensor lands on the first device, which then spills to host memory over PCIe. Verify placement by reading actual GPU memory, never by assuming:

for c in card1 card2; do
  printf "%s vram %s gtt %s\n" "$c" \
    $(cat /sys/class/drm/$c/device/mem_info_vram_used) \
    $(cat /sys/class/drm/$c/device/mem_info_gtt_used)
done

Placement must not go on the service command line: command-line arguments beat the environment, so a --device flag there silently overrides every preset.

Server slots and context

--parallel sets llama-server's slot count, and slots divide ctx-size rather than share it: -c 8192 -np 4 reports n_ctx_slot = 2048. A multi-slot preset therefore cuts the context any single request can use.

The default is 1, so a model gets its full declared context. Raise it only where concurrent requests are genuinely wanted, and size ctx-size for the division.

Labelled lanes

A lane is a second preset over the same weights with a context sized to a role. Sequential coordinator and worker turns do not co-reside, but llama-server reserves the full KV cache up front, so a worker that needs 8k should not declare 262144.

llama-model-add \
  --repo unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --no-download \
  --file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  --label ctx32k \
  --ctx-size 32768 \
  --reasoning-budget 16384 \
  --n-predict 24576 \
  --restart

The lane reuses the base model's directory, so the GGUF is stored once no matter how many lanes point at it.

Adding a lane to a model that is already registered

Pass --no-download with the file's basename; no directory prefix, because the weights are already in managed storage. Nothing is copied or moved, since the target path already exists.

On a new lane, set every value you care about. Omitted settings fall back to defaults, not to the base section:

parallel = 1          # the default
ctx-size = 32768
cont-batching = true

No reasoning-budget and no n-predict are written, so the lane's budgets are unbounded, which defeats the point of sizing the lane to a role. Pass both budgets explicitly.

On a lane that already exists, omitted settings are inherited from that section, so re-running a short form is idempotent and safe.

Without --restart the preset is written but the running server keeps its old registry, and the lane will not appear in /v1/models until the service restarts.

Remove one with the section name, not the served ID:

llama-model-remove 'ORG/REPO:Q4_K_M-ctx32k' --restart

Without --delete-files this leaves the shared weights in place.

The section name is not the model ID. llama.cpp renames a labelled section: ORG/REPO:Q4_K_M-ctx32k is served as ORG/REPO:CTX32K, and a request for the section name is rejected with model not found. llama-model-add prints the served ID as Served as:; use that value in models.json.

Leave the full-context lane at n-predict = -1 and reasoning-budget = -1 so coding turns are bounded only by the context window.

Qwen thinking profiles

Apply the native-context profile for routine use:

llama-model-add \
  --repo bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF \
  --no-download \
  --file ThinkingCap-Qwen3.8-27B-Q4_K_M.gguf \
  --profile qwen-thinking-native \
  --restart

The profile sets:

parallel = 1
ctx-size = 262144
temperature = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
reasoning-budget = 65536
n-predict = 98304

The native profile disables static RoPE scaling. Its 98,304-token total completion limit reserves up to 65,536 reasoning tokens plus 32,768 final tokens, leaving at least 163,840 tokens for input when both limits are reached.

The qwen-thinking-1m profile remains available as an explicit opt-in. It uses static YaRN to extend the native 262,144-token context to 1,000,000 tokens, with a 262,144-token reasoning budget and 393,216-token total completion limit. This requires substantially more KV-cache memory and may reduce quality on short contexts.

For non-thinking mode under the native profile, send:

{
  "chat_template_kwargs": {"enable_thinking": false},
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repeat_penalty": 1.0,
  "max_completion_tokens": 32768
}

Pi integration

pi/qwen-sampling-profiles.ts applies Qwen's recommended sampler to Qwen models from the local-llm, local-llm-igpu and local-strata providers. Pi's thinking state selects the mode:

Thinking off   temperature 0.7, top-p 0.8, presence-penalty 1.5
Thinking on    temperature 1.0, top-p 0.95, presence-penalty 0.0

The extension sets sampler values only, never token budgets. Sampling is a property of the model family and is the same on every lane. Token budgets are a property of the preset a model is served under: a coordinator preset at ctx-size 32768 and a full-context preset at 262144 need different n-predict and reasoning-budget, and both are already set in presets.ini. An extension that also sent budgets would push one context's numbers onto every lane, which is what it did before this was split.

So the ownership is:

Setting	Owner
temperature, top-p, top-k, min-p, penalties	this extension
max_tokens per model	models.json
n-predict, reasoning-budget, ctx-size	presets.ini

Configure each Qwen model in ~/.pi/agent/models.json with reasoning: true, a contextWindow matching its preset's ctx-size, a maxTokens matching its preset's n-predict, and:

{
  "thinkingLevelMap": {
    "minimal": null,
    "low": null,
    "medium": null,
    "high": "high",
    "xhigh": null,
    "max": null
  },
  "compat": {
    "thinkingFormat": "qwen-chat-template"
  }
}

This leaves only off and high as selectable Pi thinking levels. Use Shift+Tab in the TUI or pass --thinking off or --thinking high.

Administration

llama-model-list
llama-model-check
llama-model-restart
llama-model-remove 'ORG/REPO:QUANT' --restart
llama-model-remove 'ORG/REPO:QUANT' --delete-files --restart

llama-model-check validates the preset registry against the files on disk, the served API, and Pi: a model whose preset loads an mmproj must declare "image" in its models.json input array, and one without a projector must not. Pi refuses to send an image the entry does not claim, so a stale entry silently disables vision on a model that supports it. The Pi file defaults to ~/.pi/agent/models.json and is overridden with LLAMA_PI_MODELS_FILE; the provider is matched to the lane by its baseUrl, so each lane checks its own. Models absent from models.json are not flagged.

Registration and removal lock the preset, replace it atomically, and retain five timestamped backups. With --restart, the command validates /v1/models and restores the previous preset if validation fails.

llama-model-restart restarts every installed llama-server*.service lane, discovering them rather than assuming two, and waits for each to answer /v1/models before reporting it up. A restart does not return memory held in the kernel's GPU page pool; see "Memory still used after unloading a split model" above. --list shows each lane's port, state and loaded models; naming units restarts only those. To drop models without restarting, use llama-model-unload.

The managed service is:

systemctl --user status llama-server.service
systemctl --user restart llama-server.service

The package-provided system llama.cpp.service is separate and should remain disabled.

Tests

./tests/test.sh

The test uses a temporary home directory and does not download a model.

Bootstrap integration

omarchy-dev-bootstrap clones this repository before its local-LLM module runs. That module installs llama.cpp-vulkan, calls install.sh, and enables the user service when managed model files and runtime presets exist.

Future improvement

Add a portable model manifest so a new machine can reproduce the installed model set and per-model settings without committing the generated runtime preset.