Vigil

• How it works
  • What a state means
  • Runaway processes
  • Crash evidence
• Install
  • Acknowledging a known problem
• Configuration
• Share it securely (Cloudflare Tunnel + Access)
• Security
• Dependencies
• Not in this version

Vigil watches the machine it runs on. Once a minute it reads the host (CPU, memory, disks, temperatures, GPUs, network, systemd units, containers, local HTTP endpoints), turns each thing into a service with a state, and shows the result on a web page. The model is Checkmk's: every service is OK, WARN, CRIT or UNKNOWN, services are found by discovery rather than listed by hand, and anything that disappears turns CRIT instead of silently vanishing.

It is read-only. It never restarts, stops or changes anything it monitors.

How it works

 collectors (agent sections)          checks (pure functions)          SQLite              web UI
 /proc, /sys, df, ss,          ──►    cpu_load, memory, filesystem, ─► services, current ─► 127.0.0.1:3100
 systemctl show, tailscale,           temperature, gpu, interface,     state, events,       │
 docker ps, HTTP probes               systemd_*, http, tunnel, ...     metric history       ▼
                                                                                 Cloudflare Tunnel + Access
                                                                                 https://monitor.<domain>

• Collectors (src/collect.ts) gather one cycle of raw data. Each runs in parallel; one that fails leaves its section out, its checks go UNKNOWN, and the "Vigil collector" service names it.
• Discovery (src/discovery.ts) turns that data into services: one per real filesystem (btrfs subvolumes and bind mounts collapse to the shortest mount; mounts under filesystemIgnore prefixes, by default removable media under /run/media/, are skipped), per temperature sensor, per AMD GPU, per network interface that was up (except Docker's veth*, docker0 and br-<id> interfaces, which come and go with containers), per enabled systemd unit that was running (user and system manager; oneshots are judged by their last result, and units started on demand by a D-Bus name or a socket, timer or path unit are OK while inactive after a successful run), per running rootless Docker container that carries the label vigil.monitor=true (start a long-lived container with --label vigil.monitor=true; test and sandbox containers stay out), and per configured HTTP endpoint. New services are added; old ones are never dropped automatically.
• Checks (src/checks.ts) turn data into a state, a one-line summary and metrics. They are pure functions, tested on captured real output in test/fixtures/.
• Store (src/store.ts) keeps the services, their current state, every state change (the Events tab) and metric history for retentionDays (default 7).
• Web UI (public/) shows the state counts, problems first, every service grouped, and a graph per metric with its warn/crit lines. It refreshes every 30 s.

What a state means

Service	WARN	CRIT
CPU load	15 min load ≥ 1.0 per thread	≥ 2.0 per thread
Memory	≥ 80 % used (MemTotal − MemAvailable)	≥ 90 %
Swap	≥ 50 %	≥ 80 %
Filesystem	≥ 80 % (same ratio as df Use%)	≥ 90 %, or no longer mounted
Temperature, GPU	10 °C below the sensor's own critical point, else 80 °C	at the critical point, else 90 °C
GPU	(temperature only; full VRAM is normal for LLM serving)	card gone
Interface	new rx/tx errors	link down
Systemd unit	activating/reloading, or restarted in the last 15 min	inactive, failed, not found; oneshot failed
Failed units (summary)	a failed unit in that manager not matched by failedIgnore
HTTP endpoint	a status listed in warn (e.g. 503 while loading), slow	refused, timed out, unexpected status
Cloudflare tunnel		no edge connection, metrics unreachable
Tailscale		backend not Running, or offline
Listening sockets	a TCP listener on all interfaces not in exposureAllow
Container		not running, or gone two cycles in a row
Kernel crash log		any record in the pstore archive
Previous boot	it ended without a clean shutdown or reboot
Vigil collector	a collector failed, or a cycle took ≥ 10 s
Runaway processes	a process averaged at least 20 % of one core for 60 min	(never)

A check that cannot get its data, or crashes, is UNKNOWN.

Runaway processes

CPU load and utilisation describe the whole machine, so one process stuck in a loop at 30 % of a core stays invisible on a 24-thread box. This service looks at each process instead. Every cycle it reads utime + stime from /proc/<pid>/stat, and it is WARN while any process has averaged at least levels.runawayCpuPct (20) % of one core over the last levels.runawayMin (60) minutes. The summary names the command line, PID, share, running time and parent, so the culprit can be found from the phone.

• Only the process's own CPU counts, not its children's, so a long build's make is not flagged for its compilers.
• A process is its PID plus its start time, so a reused PID never inherits another process's history. Kernel threads and zombies are skipped.
• A verdict needs a full window of samples. After a Vigil restart the service is OK for the first runawayMin minutes.
• A long job that is meant to run (a synthesis, a benchmark) can be acknowledged, or its command name (comm, at most 15 characters, as ps -o comm shows it) added to runawayIgnore. An acknowledgement ends when the set of flagged processes changes.
• Vigil only reads /proc; it never signals, renices or kills a process.

Crash evidence

Two services record what a whole-system freeze or kernel panic leaves behind. They look backwards, so they turn up after the reboot, not during the crash.

• Kernel crash log is CRIT while /var/lib/systemd/pstore/ (pstoreDir) holds any entry. systemd-pstore.service copies the kernel's crash log there on the boot after a panic; the summary gives the time and the first panic, BUG or lockup line of dmesg.txt. Vigil never deletes records, so the service stays CRIT until you read them and remove them as root. Records the monitoring user cannot read are still CRIT and say "unreadable". The check needs the kernel to write to the EFI variable store: add efi_pstore.pstore_disable=N to the kernel command line (the kernel default is disabled; cat /sys/module/efi_pstore/parameters/pstore_disable shows Y when it is off). To get a panic instead of a silent freeze, also set kernel.softlockup_panic=1, kernel.hardlockup_panic=1 and kernel.panic=30.
• Previous boot is WARN when the boot before this one ended without a clean shutdown or reboot, with the time of its last journal entry; it stays WARN until the next boot. A boot counts as clean when the system manager (PID 1) logged "Stopped target Multi-User System." and its own message is the last entry in the journal. The result is cached per boot ID, so it costs two journalctl calls per boot. It needs a persistent journal readable by the monitoring user.

Install

cd ~/Documents/10-repos/11-gitRepo/vigil
bun test                                   # 116 tests
bun src/main.ts once                       # one cycle, prints every service
systemctl --user link "$PWD/deploy/vigil.service"
systemctl --user enable --now vigil
loginctl enable-linger "$USER"            # keep running when logged out

The page is then at http://127.0.0.1:3100. Localhost needs no login.

Other commands: bun src/main.ts services prints the stored state; bun src/main.ts forget <id> removes a service you retired on purpose (ids are shown in the service's detail view, e.g. systemd_user:foo.service).

Acknowledging a known problem

Once a WARN or CRIT has been investigated and its root cause is known, acknowledge it:

bun src/main.ts ack previous_boot --cause "kernel soft-lockup panic" --evidence ~/freeze-captures/pstore-20261019

The service then reads OK, with its summary prefixed Acknowledged WARN: and the cause in its details. The acknowledgement ends by itself when the state changes (back to OK, or to a different severity) or when the check reports a new incident. previous_boot identifies an incident by boot id and kernel_crash by its newest record, so the next unclean boot or panic shows again. bun src/main.ts unack <id> ends one early, and the problem shows again at once.

The same can be done from the page: a service's detail view has an Acknowledge button with a cause field, and an acknowledged service has Remove acknowledgement. The desktop page (localhost) always shows them. Through monitor.<domain> they appear only for an Access login listed in ackEmails; everyone else Access lets in sees the page read-only. The acknowledgement records who made it: localhost, or the Access email.

Every acknowledgement is kept, including after forget, as a record of the problem: the state, summary and details Vigil showed, the cause, the evidence, who acknowledged it, and when and why it ended. bun src/main.ts acks prints it; /api/acks serves it as JSON.

Configuration

Optional, at ~/.config/vigil/config.json (VIGIL_CONFIG overrides the path). Top-level keys replace the defaults in src/config.ts; levels and access merge key by key. See deploy/config.example.json.

• http: endpoints to probe, each { name, url, expect: [codes], warn?: [codes], slowMs? }. Redirects are not followed, so a front door behind Access is checked by expecting 302.
• exposureAllow: ports that may listen on all interfaces.
• failedIgnore: failed units the failed-units summary skips, each { unit, result? }. With result, the unit is skipped only when systemd recorded that Result (systemctl show -p Result <unit>), so the same unit failing another way still warns; the summary names what it skipped. Example: { "unit": "fprintd.service", "result": "core-dump" } for fprintd's crash when a fingerprint prompt times out.
• ackEmails: Access emails that may acknowledge from the page. Empty by default, so the remote page is read-only until it is set.
• runawayIgnore: command names never flagged as runaway processes; levels.runawayCpuPct and levels.runawayMin set the share and the window.
• filesystemIgnore: mount-point prefixes discovery skips, default ["/run/media/"], where udisks automounts removable media. A stick that is unplugged on purpose would otherwise stay as a service and go CRIT. To watch a permanently attached drive, mount it through fstab elsewhere, or set the list to [].
• access.aud: the Cloudflare Access application's AUD tag. While empty, Vigil answers localhost only.

Share it securely (Cloudflare Tunnel + Access)

Same front door as the interpreter: the existing tunnel gets one more hostname, and Cloudflare Access asks for a one-time PIN before anyone reaches the page. Vigil also checks the Access token itself, so the page stays closed even if the Access application were deleted by mistake.

1. Access application first. Zero Trust → Access → Applications → Add → Self-hosted. Domain monitor.<domain>, policy Allow → Emails → your address, login method One-time PIN only, long session.
2. Copy the application's AUD tag (Overview → Application Audience) into ~/.config/vigil/config.json as access.aud, with access.teamDomain set to your <team>.cloudflareaccess.com. Restart: systemctl --user restart vigil.
3. Tunnel route. cloudflared tunnel route dns <tunnel> monitor.<domain>, add the ingress rule from deploy/cloudflared-ingress.yml above the final 404 rule in ~/.cloudflared/config.yml, then restart the tunnel service.
4. Check: an anonymous curl -sI https://monitor.<domain> must answer 302 to cloudflareaccess.com; a local curl -s -H 'Host: monitor.<domain>' 127.0.0.1:3100/api/overview must answer 403.

Security

• Listens on 127.0.0.1 only; the tunnel is the only way in from outside.
• Any request whose Host is not localhost needs a valid Access JWT (RS256 signature from the team's published keys, issuer, audience, expiry). That also defeats DNS-rebinding pages.
• Strict Content-Security-Policy, no inline scripts, all data rendered as text.
• The only writes are POST /api/ack and POST /api/unack, and they change Vigil's own database, never the system. Because localhost needs no login, any website open in the desktop browser could try to post to 127.0.0.1:3100, so a write must have a JSON body, an Origin equal to its Host, and Sec-Fetch-Site: same-origin when the browser sends one. A JSON body from another site needs a CORS preflight, which Vigil never answers, so the browser does not send it. Every other method and path is GET only.
• External programs run through execFile with argument arrays, never a shell.
• Read-only towards the system: systemctl is only called with show, list-unit-files and --failed, and processes are only read from /proc. GPU counters are not read from a runtime-suspended card, since reading them would wake it.

Dependencies

What	Version proven on	Used for
Bun	1.4.0	runtime, bun:sqlite, tests
Linux, systemd	7.2 / systemd 261	/proc, /sys/class/{hwmon,drm,net}, systemctl --user and system
coreutils df, iproute2 ss	Arch current	filesystems, listening sockets
tailscale CLI	1.102.3	Tailscale state (service is UNKNOWN without it)
docker CLI, rootless daemon	socket /run/user/<uid>/docker.sock	container states (dockerHost)
journalctl, systemd-pstore	systemd 261	previous-boot end, kernel crash records (/var/lib/systemd/pstore)
cloudflared	tunnel metrics on 127.0.0.1:20241	tunnel readiness (tunnelMetricsUrl)
amdgpu driver	R9700 (1002:7551), 890M (1002:150e)	GPU counters; other vendors are not read yet
Cloudflare Zero Trust	free plan	Access application + one-time PIN login
TypeScript, @types/bun	dev only	bunx tsc -p . typecheck

No runtime npm packages.

Not in this version

Notifications and downtimes, more than one host (the collector/check split is shaped for a remote agent later), checks that need root (SMART), pending package updates.