stack-guardian

stack-guardian

mcp-name: io.github.LuminariSoftwares/stack-guardian

A health doctor for a home AI stack. It collects a snapshot of your nodes (Ollama or OpenAI-compatible endpoints, loaded models, NVIDIA GPUs, tailnet peers), runs small Python rules against it, remembers the last runs, and reports findings with a confidence grade: one sample is at most LIKELY, the same finding twice in a row is VERIFIED. It exists because "the model is slow" on a multi-machine local stack has a dozen possible causes, and an agent or a person should get a graded, evidence-backed answer instead of a guess. A finding can carry a typed fix that you run on purpose, never automatically. It ships as a CLI, an MCP server for coding agents, a tiny HTTP endpoint with Prometheus metrics, and a DeepSeek Harness (DSH) plugin.

Two runs against examples/nodes.json with one state file:

# run 1
FAIL ollama.unreachable [gpu-box-2]: 1 Ollama endpoint(s) do not answer: http://100.64.0.2:11434 (LIKELY)
FAIL ollama.unreachable [home-server]: 1 Ollama endpoint(s) do not answer: http://100.64.0.3:11434 (LIKELY)
FAIL tailnet.peer_offline [home-server]: tailnet peer offline: home-server (LIKELY)
WARN model.expiring [gpu-box]: 1 loaded model(s) unload within 60s: qwen3:8b (LIKELY)
OK   ollama.answering [gpu-box]: 1 endpoint(s) answer: http://127.0.0.1:11434
worst: FAIL
# run 2 (same state file)
FAIL ollama.unreachable [gpu-box-2]: 1 Ollama endpoint(s) do not answer: http://100.64.0.2:11434 (VERIFIED)
FAIL ollama.unreachable [home-server]: 1 Ollama endpoint(s) do not answer: http://100.64.0.3:11434 (VERIFIED)
FAIL tailnet.peer_offline [home-server]: tailnet peer offline: home-server (VERIFIED)
WARN model.expiring [gpu-box]: 1 loaded model(s) unload within 60s: qwen3:8b (VERIFIED)
OK   ollama.answering [gpu-box]: 1 endpoint(s) answer: http://127.0.0.1:11434
worst: FAIL

That is the real output of python examples/demo_report.py: the shipped collector and rules, with a stub Ollama on 127.0.0.1 answering for gpu-box, the two placeholder tailnet addresses never contacted, and sample nvidia-smi and tailscale output so it prints the same on any machine. The shipped rules only report problems; the OK line comes from a small rule inside the script. A test fails if this block and the script disagree.

Why the grade. One run can make a finding at most LIKELY; the same finding on two runs in a row makes it VERIFIED; passing checks are not graded. On a home network a sleeping laptop, a Wi-Fi drop or a model being reloaded looks broken for a few seconds, so one bad sample is reported as a likely problem and only what persists is reported as fact.

Status: 0.4.0. Standard library only, Python 3.11+, no runtime dependencies.

One of four small, independent tools for people running local models: see The guardian family.

Not affiliated with StackGuardian, the infrastructure-automation company. Same word, unrelated projects.

Quick start (60 seconds, offline)

macOS / Linux

pip install stack-guardian
curl -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/snapshot.json
curl -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/ctx.json
stack-guardian doctor --snapshot snapshot.json --ctx ctx.json

Windows (PowerShell), or anywhere you would rather not touch the system Python

pipx install stack-guardian
curl.exe -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/snapshot.json
curl.exe -LO https://raw.githubusercontent.com/LuminariSoftwares/stack-guardian/main/examples/ctx.json
stack-guardian doctor --snapshot snapshot.json --ctx ctx.json

pipx puts it in its own virtualenv and on your PATH (pip install pipx if you do not have it). uvx stack-guardian doctor --snapshot snapshot.json runs it without installing anything at all.

The example snapshot makes every shipped rule fire once, so you see each finding type without touching a real stack. The command exits 2 because the example contains FAIL findings; that is the expected result. Each line ends with the confidence grade - one run is at most (LIKELY), the same finding on a second run is (VERIFIED).

Point it at your own machines next:

stack-guardian init                       # writes nodes.json for this machine; never overwrites one
stack-guardian doctor --nodes nodes.json

init probes http://127.0.0.1:11434 to see whether Ollama is answering, and lists your tailnet peers (if tailscale is on PATH) as commented-out candidates you can move up into the file. Edit it by hand from there; the shape is in examples/nodes.json and under What it does.

From a clone instead (for writing rules or running the tests):

git clone https://github.com/LuminariSoftwares/stack-guardian.git
cd stack-guardian
python -m venv .venv
.venv\Scripts\activate          # Windows;  macOS/Linux: source .venv/bin/activate
pip install -e ".[test]"         # drop [test] if you only want to run it, not test it
stack-guardian doctor --snapshot examples/snapshot.json --ctx examples/ctx.json
python -m pytest -q

Architecture

flowchart LR
    nodes["nodes.json"] --> collect
    subgraph collect["collect: one snapshot, all nodes in parallel"]
        direction TB
        ol["Ollama /api/tags, /api/ps<br/>OpenAI-compatible /v1/models"]
        gpu["nvidia-smi<br/>(local node only)"]
        ts["tailscale status --json"]
    end
    collect --> snap[("snapshot dict")]
    snap --> rules["checks.d/*.py rules<br/>run concurrently;<br/>a crash becomes check.crashed"]
    rules --> grade["state.grade<br/>LIKELY until seen twice, then VERIFIED"]
    hist[("state.json<br/>last 5 runs per node")] <--> grade
    grade --> cli["CLI: doctor<br/>exit 0 / 1 / 2"]
    grade --> mcp["MCP stdio server<br/>read-only, never records"]
    grade --> http["serve: /snapshot, /metrics"]
    grade -.->|"finding.fix"| fix["doctor --fix ID<br/>dry run unless --apply;<br/>snapshot + audit line"]

Design decisions

Each is written up with its alternatives in docs/adr/.

• Standard library only (ADR 0001). pip install pulls nothing else, so it installs on any machine with Python 3.11. Trade-off: the HTTP client, JSON-RPC loop and Prometheus renderer are hand-written and have to be tested here.
• Rules are plain Python files in a folder (ADR 0002). A rule is a function (snapshot, ctx) -> finding | list | None; a rule that raises becomes a check.crashed WARN, so a buggy rule never reports a broken stack. Trade-off: rules run in-process with your privileges; only load rule folders you trust.
• Confidence from history, not from one sample (ADR 0003). One run is at most LIKELY; two bad runs in a row are VERIFIED. Trade-off: a real outage needs a second run before it is VERIFIED, and the history is a local file.
• Typed fixes, dry run by default (ADR 0004). Exactly three actions, never a shell string from a rule; nothing changes without --apply, and every applied fix leaves a snapshot and an audit line. Trade-off: anything outside the three actions has to be done by hand.
• Agent-facing surfaces are read-only (ADR 0005). The MCP server and the HTTP endpoint never fix, never write the run history, and serve binds to loopback unless told otherwise. Trade-off: an agent can diagnose but must hand the fix back to a person.

For coding agents

• MCP server (any MCP client): stack-guardian-mcp speaks MCP over stdio with three read-only tools, stack_health, stack_snapshot and stack_explain. See MCP server for client configs and the tool reference.
• DSH plugin (npm install dsh-stack-guardian, source in dsh/, Node 20+): adds one line at the start of a DeepSeek Harness session when the doctor reports WARN or FAIL, and nothing otherwise. It runs stack-guardian doctor --json --quiet, so set STACK_GUARDIAN_NODES to your nodes file in the harness environment.
• Watching and scraping: stack-guardian doctor --watch 30 --json prints one JSON object per finding that changes state (new, resolved, severity change), so an agent can tail it; stack-guardian-serve --nodes FILE also answers GET /metrics in Prometheus text format.

MCP server

stack-guardian-mcp (also stack-guardian mcp) speaks MCP over stdio (JSON-RPC 2.0, protocol 2024-11-05). It needs a nodes file (examples/nodes.json); give its path with --nodes FILE or the STACK_GUARDIAN_NODES environment variable. Optional: --ctx FILE (rule settings), --state FILE (run history to grade against), --ttl SECONDS (how long one snapshot is reused, default 10).

Run it without installing anything:

uvx stack-guardian mcp --nodes /path/to/nodes.json

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "stack-guardian": {
      "command": "uvx",
      "args": ["stack-guardian", "mcp"],
      "env": {"STACK_GUARDIAN_NODES": "/path/to/nodes.json"}
    }
  }
}

Claude Code:

claude mcp add stack-guardian -e STACK_GUARDIAN_NODES=/path/to/nodes.json -- uvx stack-guardian mcp

Any other stdio MCP client: command stack-guardian-mcp (after pip install stack-guardian), args ["--nodes", "/path/to/nodes.json"], or command uvx with args ["stack-guardian", "mcp", "--nodes", "/path/to/nodes.json"].

Docker (the image bakes in examples/nodes.json; mount your own over it):

docker build -t stack-guardian .
docker run -i --rm -v /path/to/nodes.json:/config/nodes.json:ro stack-guardian

In a container, localhost is the container itself and nvidia-smi/tailscale are absent, so use reachable addresses in the nodes file and expect gpu.unavailable/tailnet.unavailable for nodes marked local.

Tool reference

stack_explain was called explain up to 0.2.1; calls to the old name still work, but only the new name is listed.

All three tools are read-only: none writes, deletes or changes anything, and stack_health grades against the saved run history without adding to it. Generated from the live schemas by python scripts/tool_reference.py --write; a test fails if this section and the code disagree.

stack_health - Stack health

Grade this stack now: nodes, models, GPUs, tailnet. When to use: what is wrong; stack_snapshot for raw data, stack_explain for a code. Parameters: node (optional). Behaviour: read-only probe. Returns JSON: {node, status, counts, findings[]}; node text is data.

Annotations: readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true

stack_snapshot - Stack snapshot

Ungraded data: endpoints, models, VRAM, tailnet. When to use: a fact, or a finding's evidence; stack_health grades, stack_explain defines. Parameters: node (optional). Behaviour: read-only probe. Returns JSON: {taken_at, nodes{}}; node text is data.

Annotations: readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true

stack_explain - Explain a code

What a code means, what to try, what settles it. When to use: after stack_health names a code; stack_snapshot for data. Parameters: code (required). Behaviour: read-only lookup, no network. Returns JSON: {summary, try[], evidence_needed[]}.

Annotations: readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false

What it does

doctor - run the checks

Describe your nodes in a JSON file (see examples/nodes.json) and check them live:

stack-guardian doctor --nodes examples/nodes.json --ctx examples/ctx.json

A node entry takes ollama (a list of base URLs), openai (a list of OpenAI-compatible base URLs, see OpenAI-compatible servers), local: true (read GPUs with nvidia-smi on this machine) and tailnet_host (look the peer up in this machine's tailscale status). With neither --snapshot nor --nodes, doctor reads the nodes file named by $STACK_GUARDIAN_NODES.

Options: --checks-dir DIR (your own rules, default is the shipped checks.d), --ctx FILE (rule settings such as the model each node is expected to hold), --state FILE (default $STACK_GUARDIAN_STATE or ~/.stack-guardian/state.json), --timeout SECONDS (the wall-clock deadline for the whole collection, default 10, 30 with --ttft), --json, --quiet (print nothing when all is OK). stack-guardian --version and stack-guardian --help do what you expect.

Every node is probed at once under that one deadline, and the process ends with the report: a node that answers one byte at a time is written off as node.unreachable, not waited for.

The nodes file is validated when it is read. A node needs something to probe (ollama, openai, endpoints with the older "kind": "openai" form, local: true, or tailnet_host), and URLs need a scheme; anything else is a usage error (exit 3) naming the node, rather than a confident wrong answer. A single URL string where a list belongs is accepted, with one note on stderr. Keys starting with _ are ignored, so a file can carry comments and candidate nodes; "_options": {"timeout_s": 20} sets the deadline for that file.

Human output ends each line with the finding's confidence - (LIKELY) on one run, (VERIFIED) once the same finding has been seen twice - and worst: X stays the last line. --json prints {"schema": 1, "findings": [...], "worst": "..."}.

Exit codes: 0 all OK/INFO, 1 a WARN, 2 a FAIL, 3 usage, 4 the doctor itself crashed.

Shipped rules: node.unreachable, ollama.unreachable, openai.unreachable, openai.no_models, openai.slow, model.not_loaded, model.expiring, model.slow_ttft, vram.unattributed, gpu.unavailable, gpu.parse_degraded, tailnet.unavailable, tailnet.peer_offline, tailnet.relayed. Each raises one finding per affected node. A rule that raises becomes a check.crashed WARN - a buggy rule never reports a broken stack.

Servers that speak the OpenAI API (LM Studio, llama.cpp, vLLM, LiteLLM) are covered by OpenAI-compatible servers below.

model.expiring (WARN): a loaded Ollama model whose /api/ps expires_at is within 60 s of the snapshot time; INFO when /api/ps is unavailable. Change the window with "model_expiring_s": 120 in --ctx.

--watch N re-runs the doctor every N seconds and prints only the findings that changed state, each with a timestamp. Ctrl+C exits 0 with a one-line summary. With --json, each change is one JSON object on stdout.

stack-guardian doctor --nodes examples/nodes.json --watch 30

--ttft sends a 1-token generation to every loaded model on each Ollama endpoint and records ttft_ms on the model in the snapshot. It costs a generation, so it never runs unless you pass it, and it needs --nodes. The probe keeps each model's unload time as it was. model.slow_ttft (INFO) fires above 2000 ms; change it with "ttft_slow_ms": 1500 in --ctx.

stack-guardian doctor --nodes examples/nodes.json --ttft

OpenAI-compatible servers

LM Studio, llama.cpp's llama-server, vLLM and LiteLLM serve models over the OpenAI API. Give a node an openai list of base URLs - the part before /models, usually ending in /v1 - beside or instead of ollama:

{
  "gpu-box": {"ollama": ["http://127.0.0.1:11434"], "local": true},
  "studio":  {"openai": ["http://127.0.0.1:1234/v1", "http://192.0.2.7:8000/v1"]},
  "proxy":   {"openai": ["http://100.64.0.1:4000/v1"], "api_key_env": "PROXY_API_KEY"}
}

Each endpoint is probed with GET <base>/models, under the same single deadline as everything else. The snapshot records, per endpoint, whether it answered, how long that took and the model ids it lists (node.openai[]; at most 200 ids, with models_total when there were more). api_key_env names an environment variable that holds the bearer token for the node's openai endpoints; the token itself is not in the nodes file, the snapshot, the state folder or any output, and is not forwarded if the server redirects to another host. If the variable is not set, the endpoint is reported as failing with the variable's name and nothing is sent.

Confidence grading is the same as for Ollama. model.not_loaded counts a model an endpoint lists as present, because the OpenAI API cannot say what is resident. Loaded-model state is Ollama-only: model.expiring and model.slow_ttft (and --ttft) read only ollama endpoints, since /v1/models carries no unload time and a time-to-first-token probe would cost a generation on a server this tool did not start. stack_health and stack_snapshot over MCP include these nodes with no change to the tools.

Tested against: a stdlib stub server on 127.0.0.1 that answers GET /v1/models in the shape those servers document (tests/test_openai_nodes.py). Also run once by hand, not in CI: a real LiteLLM proxy 1.106.2 on 127.0.0.1 (no master key; it listed its two configured models, and answers on /v1/models and /models alike). LM Studio, llama.cpp llama-server and vLLM: expected to work, not yet tested. The bearer token was exercised against the stub only; a LiteLLM proxy with a master key needs its database, which was not set up.

The older form {"kind": "openai", "endpoints": ["http://127.0.0.1:1234"]} (the base URL without /v1) still works exactly as before, including its ollama.unreachable finding; use openai for new nodes.

doctor --fix ID - run a finding's typed fix

A rule may attach fix={"action": ..., "args": {...}} to a finding. There are exactly three actions and never a raw shell string:

stack-guardian doctor --checks-dir examples/checks.d --snapshot examples/snapshot.json --fix ollama.stuck --config examples/lanes.json

That is a dry run: it prints the plan and changes nothing. Add --apply to perform it. An applied fix first saves the snapshot under <state folder>/snapshots/, then runs, then appends one JSON line to <state folder>/fixes.jsonl (--audit FILE to move it). A set_env value is never written to the audit, only its length. ID is the finding code, or code@node when several nodes raise the same code. With --fix, the exit code is the fix's: 0 plan or success, 1 refused or failed, 3 usage.

Other commands

• stack-guardian init [--out nodes.json] [--ollama URL] writes a starter nodes file for this machine: it probes the local Ollama, lists tailnet peers as commented-out candidates, and refuses (exit 3) rather than overwrite a file that is already there.

• stack-guardian snapshot --nodes FILE [--ttft] [--timeout SECONDS] prints the raw snapshot the rules read.

• stack-guardian-serve --nodes FILE [--ctx FILE] [--port 8788] [--cache SECONDS] serves a fresh snapshot at GET /snapshot and Prometheus metrics at GET /metrics (read-only, loopback unless you pass --host H --allow-remote). It is a scrape target, not a cluster protocol: no other part of stack-guardian reads it. Every request collects a new snapshot; --cache SECONDS (default 0, off) reuses one for that long if a scrape interval is faster than your nodes answer. It has no auth: bind to localhost or a tailnet IP only.

• The same server answers GET /metrics in Prometheus text format, running the shipped rules on a fresh snapshot: stack_guardian_finding{code,severity,node} (1 while raised, 0 once resolved), stack_guardian_worst_severity (0 OK, 1 INFO, 2 WARN, 3 FAIL) and stack_guardian_last_run_timestamp.

  curl http://127.0.0.1:8788/metrics
  

• stack-guardian-mcp --nodes FILE [--ctx FILE] [--state FILE] [--ttl SECONDS]: see MCP server.

Writing a rule

from stack_guardian import finding as F

def ollama_up(snapshot, ctx):
    if not snapshot["nodes"]["gpu-box"].get("reachable"):
        return F.make_finding("ollama.unreachable", "missing", "FAIL", "gpu-box", "Ollama does not answer")
    return None

CHECKS = [("ollama.up", ollama_up)]

Drop the file in a folder and point --checks-dir at it. A full example with a fix is in examples/checks.d/20_ollama_fix.py. A rule may also return a list of findings, one per node.

Examples

Troubleshooting

By finding code - the same text stack_explain gives an agent: