caliper

Caliper

Know if your agent skill actually works.

Your skill worked when you tried it. Will it work the next nine times? After the next model update? When another skill competes for the same prompt?

Quick start  •  How it works  •  Docs  •  Examples

Why Caliper

Real installs Your skill is installed where the agent looks, never pasted into the prompt.

Activation scoreboard Did the agent pick your skill? Scored apart from whether it worked.

Neighbourhoods Declare competing skills and assert which one should win each prompt.

Ablation --ablate re-runs without the skill. Same score? It isn't earning its context.

Honest numbers Single-run success rate first. pass@k flatters flaky skills.

Four agents Claude Code, Codex, Pi, Hermes. One spec, pick the engine at run time.

Quick start

🤖 In your agent ⌨️ In your terminal

npx skills@latest add edonadei/caliper

# then, in your agent:
/grill-skill ./inbox-triage/SKILL.md

grill-skill interviews you, writes the spec, and runs it with and without your skill.

pipx install caliper-eval
SPEC=inbox-triage/inbox-triage.eval.yaml

caliper run $SPEC --ablate inbox-triage
caliper run $SPEC
caliper compare \
  .caliper/results/inbox-triage/RUN_ID.json \
  inbox-triage

Write a spec

A spec is a .eval.yaml file next to your skill. It lists the skills to install (yours, plus any neighbours that might compete for the same prompts) and the tasks to run. Each task is a prompt, optional setup:, and at least one check:

• expect: is graded by an LLM judge.
• assert: runs locally as Python, in the attempt's workdir, with a 30-second

limit. One that runs longer has no verdict, rather than failing the task.

• activates: asserts which skills the agent chose to load. It needs no judge,

so it costs a fraction of a graded task.

# inbox-triage.eval.yaml
skills:
  - ./SKILL.md                        # the skill under test
  - ../calendar-scheduler/SKILL.md    # a neighbour it might steal work from

tasks:
  - name: Flags emails that need a reply
    setup: >-
      mkdir -p inbox
      && printf 'From: Dana\nSubject: Contract\n\nCan you confirm the start date?\n' > inbox/1.eml
      && printf 'From: Weekly Digest\nSubject: 10 links you missed\n\n...\n' > inbox/2.eml
    prompt: "Triage my inbox."
    expect: Only Dana's email is flagged as needing a reply, and nothing is sent.

  - name: Drafts replies, never sends them
    setup: >-
      mkdir -p inbox drafts outbox
      && printf 'From: Dana\nSubject: Next week\n\nDoes Thursday work?\n' > inbox/1.eml
    prompt: "Reply to Dana and tell her Thursday works."
    assert: |
      from pathlib import Path
      assert any(Path("drafts").iterdir()), "no draft written"
      assert not any(Path("outbox").iterdir()), "sent instead of drafting"

  - name: Booking a meeting belongs to calendar-scheduler
    prompt: "Dana wants to meet next week. Find us 30 minutes."
    activates: [calendar-scheduler]

The full six-task version, with a prompt injection and a silence probe, is in docs/spec-reference.md.

The spec never names an engine. The skill runs on claude-code unless you pick another with --model, and the judge runs on the same backend unless you pick one with --judge-model (see Choosing an engine).

Read the output

caliper run inbox-triage.eval.yaml

This is the full six-task example at the default k=3:

Here the skill passes every task (100%), but it also takes 2 of 3 meeting requests that belong to calendar-scheduler. That's a description problem, not a body problem. Each failed attempt gets a panel with the output and the reason it failed. Full results are saved as JSON under .caliper/results//.

Not sure what to put in a spec?

Start from one of the examples. Each catches a common agent failure (false success, tool misuse, runaway loops, prompt regressions, stale context treated as current), passes as-is against a bundled skill, and points at your own skill by editing a few commented lines.

Documentation

· 📐 Spec reference · Every .eval.yaml field and the judging rules 🔌 Backends · Setup per agent, --model syntax, MCP support 📊 Results · Scoring, caliper compare, the results JSON schema 📖 Glossary · Spec, neighbourhood, activation, ablation… 🧭 Design decisions · Architecture Decision Records

Recommended workflow

1. Create a spec for one behavior you care about.
2. Run with --k 1 while iterating on the spec.
3. Add assert: for facts an LLM judge might guess wrong (files, JSON, command

output).

1. Before editing the skill, run once with --ablate (or

--ablate mcp:) at --k 3 and caliper compare it against a full run. A task that passes without the skill doesn't need it: sharpen it first. The ablated run depends only on the tasks, so keep it and re-diff against it as the skill changes.

1. Iterate on the skill at --k 3, and confirm a win or a regression at

--k 5 or higher before acting on it.

1. Commit the spec alongside the skill so contributors can run the same eval.

How it works

spec.eval.yaml   skills, tasks, expect: / assert:
        │
        ▼
 ┌─ Harness ──────────────────────────────────────┐
 │ runs each task k times in the agent            │
 │ (Claude Code · Codex · Pi · Hermes)            │
 │ every attempt: fresh home, empty workdir       │
 └────────────────────────────────────────────────┘
        │  one transcript per attempt
        ▼
 ┌─ Judge ────────────────────────────────────────┐
 │ expect:  an LLM reads the transcript           │
 │ assert:  Python checks the outcome             │
 └────────────────────────────────────────────────┘
        │  pass / fail per attempt
        ▼
 success rate  +  results JSON (transcripts, what loaded)

Each attempt starts clean: a temporary home with no session history and an empty working directory. Hooks run inside the attempt, under its timeout.

By default the attempt still loads your user customizations, so the score reflects the agent you actually use:

• What loads: your CLI's MCP servers, account connectors, user skills,

plugins, rules and settings.

• Name clashes: declared skills and servers win, even when ablated.
• Activation: user skills compete with declared ones and count in

activation checks.

• Built-in skills that ship with the CLI (Claude Code's claude-api) are

listed but never scored.

• Hermes keeps its neutral memory/persona policy. See

backend details.

Results are saved as JSON you can inspect and diff later. Each run records which user customizations it loaded (mcp:(not listed) when the backend can't list its MCP servers).

Portable scores

A default score depends on your setup. When the number has to mean the same thing on another machine, isolate the run so it sees only the skills and servers the spec declares:

• caliper run spec.eval.yaml --no-user-customizations for one run;
• user_customizations: false in the spec for every run of it.

Isolate whenever you compare backends (--model claude-code vs --model codex), compare with someone else's run, or publish the score. Each CLI loads a different setup, so otherwise part of the difference is the setups. caliper compare warns when two runs loaded differently.

Core concepts

Term · What it is Spec · A .eval.yaml file that describes the skills and tasks to run Backend · The CLI agent that runs the skill (claude-code, codex, pi, hermes) Judge · What decides pass/fail: an LLM reading the transcript (expect:), Python assertions (assert:), or both Success rate · The primary score: run k times, measure how often a single run works Neighbourhood · The set of skills a spec declares (skills:). All installed, none preloaded, and all assertable. This is the competition your description has to win Activation · The agent *choosing* to load a skill. Asserted with activates: and scored on its own scoreboard, separate from the success rate Ablation · Re-running the same tasks with a declared skill or mcp: server *removed* (--ablate), to prove it's doing the work Attempt · One isolated run of a single task (fresh temporary home, no session history)

The full glossary is in docs/CONTEXT.md.

Choosing an engine

The engine (backend + model) is picked at run time, not in the spec. The spec describes *what* is tested and *how* success is judged; you pick the agent that runs and grades it when you invoke Caliper. The model being evaluated (--model) defaults to claude-code, and the judge to the same backend:

caliper run my-skill.eval.yaml                          # claude-code runs and grades
caliper run my-skill.eval.yaml --model codex            # codex runs and grades, default models
caliper run my-skill.eval.yaml --model codex:gpt-5.6-sol
caliper run my-skill.eval.yaml --model pi --judge-model claude-code

Backend · Requires claude-code · Claude Code CLI installed and authenticated codex · Codex CLI (npm install -g @openai/codex), then codex login pi · pi CLI (npm install -g @earendil-works/pi-coding-agent), authenticated hermes · Hermes Agent CLI (Nous Research), authenticated, with a default model set

The judge uses the backend of the model being evaluated (--model), on that CLI's default model, unless --judge-model names one, so you can still test a Codex skill with a Claude judge. When comparing engines, pass the same --judge-model to every run: otherwise each engine grades itself, and caliper compare warns that the judges differ (ADR 0034). There's no direct-API backend: to use API billing, configure one of these CLIs with an API key.

Caliper doesn't install or update these CLIs: each one updates itself, and every saved run records which CLI version it used.

Setup details for each backend, the full --model syntax, and MCP support by backend are in docs/backends.md.

Spec format

The quick start covers the basics. A spec can also:

• pull neighbour skills from a git repo, pinned to a commit. Caliper

clones it into its own cache, so nothing is vendored into your repo:

# inbox-triage.eval.yaml
  skills:
    - ./SKILL.md
    - ../calendar-scheduler/SKILL.md
    - repo: anthropics/skills                          # a real neighbour from GitHub
      ref: dbd4588f9e1033efb41dad4bef2f7947c8993d44    # a branch, tag or commit; omit to track the default branch
      path: skills/internal-comms/SKILL.md             # also writes messages, so it competes for "Reply to Dana"

• give the agent MCP servers, local or remote (mcp:)
• pin whether your own customizations load: user_customizations: false

for a portable score from anyone, or true for a skill that needs your own connectors

• run `setup:` and `cleanup:` shell hooks in each attempt's workdir (each

killed after 600 seconds)

• start a task mid-conversation (history:): the agent resumes from

planted user and assistant turns, shell calls included, and the prompt is the next turn. A user turn of /name enters that declared skill, as if typed. setup: and the history must agree, since the agent trusts a planted output over the disk. Every backend but hermes

• extend PATH or forbid files the agent must not read (sandbox:)
• assert silence (activates: []) or a delegation chain