xienda/dsh-jev-verify

dsh-jev-verify

Jev — TypeSafe AI's System One decision model — as a first-class plugin for DeepSeek Harness (dsh).

Jev does not generate text. Given a state plus typed questions it returns typed answers with calibrated probabilities in one parallel API call (~70–500 ms published; in our runs median 266–484 ms and p95 825–1468 ms; single-run maxima reach ~1.5 s and ~5 s under a loaded network). This plugin exposes that as agent tools, adds an opt-in auto-guard (risk + loop checks), an honest local usage/quota panel, and makes sure the claims are *verified, not trusted blindly*:

Tool · What it does · Call it when jev_decision · Up to 25 typed questions (choice / score / noul) about one state, answered in parallel in one call (published ~70–500 ms; measured median 266–484 ms and p95 825–1468 ms across our runs, with single-run maxima of ~1.5–5 s depending on network load). Every answer carries a calibrated confidence and probabilities; the result also reports model, latency, token usage and estimated cost. · You need fast, repeatable verdicts instead of prose: classification/labeling, routing or triage, priority/severity/satisfaction scoring, spam/toxicity/PII checks, intent or truth checks, extracting structured tags. Batch related judgments into one call — parallel, no extra latency. jev_choose · Ranks 2–10 candidate options/approaches: each gets a fit score 0–3 and a risk noul, combined into a composite fit/3 × (1−risk); returns an ordered table, a recommended pick, and per-option latency/cost. · Several independent approaches are on the table at a fork and you want a calibrated tiebreaker before deciding. Jev scores, it never explains — the reasoning stays yours. jev_batch · Runs one set of 1–5 judgments over 1–20 texts: one real call per text (4-way concurrency by default, 6 max), returning a paste-ready markdown table sortable by a judged dimension; built-in sentiment / priority / intent / pii / spam presets, or your own questions in the jev_decision shape. · A batch of tickets/comments/logs/feedback needs labeling, classification, prioritization or PII/spam screening, or several candidates need ranking on one dimension — the tool for "many texts, same judgments"; never hand-roll one jev_decision per item. Each text is its own real call, and a failed row says 失败 with the raw error instead of guessing. jev_verify · Runs the frozen 27-question labeled benchmark (urgency, spam, toxicity, personal data, routing, intent, search type, priority, severity, satisfaction, guard verdicts) against the live API at concurrency 6 (measured 4.96 s wall for all 27 calls in 0.7.5 and 3.999 s on a re-run, versus 10.1 s serial): accuracy overall and on the high-confidence subset, median/p95/min/max latency, calibration, tokens, cost, and every mislabeled case (total plus the high-confidence ones). · Endpoint health check, model-version comparison, or a regression check — never routinely: one run is 27 real API calls (~8.7K input tokens, ≈$0.0004). jev_guard_status · Read-only audit of the auto-guard: deterministic rules and the Jev backstop counted separately (checks / jevCalls / denied / deterministicDenied / auditCalls), guarded tool names, denyThreshold, loop-check counts, remaining session budget. · Confirm the guard is armed, see how often it actually fired, or explain why a command was blocked. When the guard is off it says so instead of reporting zeros. jev_overview · Read-only snapshot of this session's Jev ledger: recent decisions and choices with confidence, latency median & p95, question-type mix, guard events, cumulative tokens and cost, plus key/guard/threshold status. · The user asks what Jev has done, blocked or spent — or you need endpoint and budget state without restarting the session. The ledger starts at plugin-instance lifetime; guard counts live in jev_guard_status. jev_usage · Locally measured Jev usage and (optional) local budgets: rolling windows (today / 7 d / 30 d / all / session), calls, input/output tokens, estimated cost, latency median & p95, question-type and per-tool mix, a projected daily burn rate, and — with quota.enabled + quota.enforce — a hard stop before the API call. · The user asks how much Jev has been used or what it cost, or you want to check a budget before a batch of judgments. Reads only local state — no API call. TypeSafe has no balance endpoint, so this is measured local usage, not provider credit.

Auto-guard mode (opt-in autoGuard.enabled): before shell-like tool calls (bash/pwsh/run_code/terminal), a free deterministic layer runs first. Seven hard rules block catastrophic, irreversible operations — recursive deletion of a filesystem, drive or directory tree, disk formatting, database destruction, credential exfiltration, force-pushed git history — and three soft rules (host restart / power-off, git-history rewrites short of a force push, always-true DELETE/UPDATE conditions) are handed to Jev (risk noul ≥ threshold ⇒ deny). Hard patterns only fire in *executable position*, so quoting or documenting a dangerous command is never blocked, and a hard match found elsewhere is downgraded to a Jev-judged hint. Jev failure is fail-open with a warning. A loop guard evaluates repeated same-tool calls for semantic stalls and injects advice instead of blocking. Every verdict is recorded in the session ledger and auditable via jev_guard_status.

Honest by design — no mock mode, no silent fallback:

• without a TYPESAFE_API_KEY, tools and guard fail with explicit setup instructions (the guard fails open with a warning — it never silently pretends to have checked);
• every jev_decision result includes the model, latency and token usage, so each call is auditable;
• jev_verify refuses to report numbers it did not measure;
• the benchmark CLI (bench/bench.mjs) is dependency-free and reproducible with any key.

What's new in 0.8.6

• Fixed a dependency mistake that could stop the host from starting (issue #1): @deepseek-ai/dsh-tools was declared as a normal dependency. In a profile using nodeLinker: hoisted + hoistPattern: ["*"], pnpm materialised a real copy under the profile's node_modules/@deepseek-ai/; the tools loader entry resolved to that copy instead of the host's own, it failed to import its peers, and startup ended in 1 required plugin did not activate. It is now an optional peer (^0.1.5-rc.2) — never installed, so it can never shadow the host. dependencies now carries only the leaf library @deepseek-ai/schemastery.
• New guard test test/manifest.test.mjs fails if any host runtime package reappears in dependencies (verified by reintroducing the bug and watching it fail). Suite: 38/38.
• Otherwise identical to 0.8.5.

What's new in 0.8.5

"Remaining" finally has an honest source — the balance you declare, minus locally measured spend — instead of a number something made up.

• A restart no longer zeroes the total. quota.persist now defaults to true: the ledger is written to $DSH_HOME/jev-usage.json, falling back to ~/.dsh/jev-usage.json when the host leaves DSH_HOME empty (that missing fallback was the root cause of the cumulative total always reading 0).
• Remaining = declared − measured. TypeSafe exposes no balance endpoint at all (/v1/usage, /v1/quota, /v1/account, /v1/balance, /v1/credits, /v1/billing, /v1/me, /v1/limits, /v1/wallet, even /v1/openapi.json all 404, and /v1/models carries no quota/credit headers), so the provider side cannot be read. Set quota.declaredBalanceUsd and quota.balanceSince (e.g. 2026-10-01) and the panel prints 剩余 = 自报 $5.00 - 本机实测 $0.000004（2026-10-01 起）; leave them empty and it says the balance was not declared rather than guessing one.
• The pill re-reads every 20 s (was 60 s), so a call you just made shows up while you are still looking at it.
• The collapsed pill leads with 剩余 once a balance is declared (Jev · 剩余 $4.9997 · 今日 3 次), and the popover renames the cumulative row when the ledger is on disk.
• New `jev_batch` — one call, one set of judgments, over a batch of texts, returned as a table. 1–20 items × 1–5 questions, one real Jev call per item (4-way concurrency by default, 6 max), returning a paste-ready markdown table; sort=auto|desc|asc|none orders rows by a judged dimension (auto only when the single question is a score, descending). Built-in sentiment / priority / intent / pii / spam presets, or your own questions in the jev_decision shape. A failed row is marked 失败 with the raw error — never guessed — and every row is metered separately (kind: "batch"), so N items really are N calls and N costs.
• The settings card gained the two declaration fields; npm test is 37/37 (the new test/batch.test.mjs adds 7 cases: one real call per item with measured cost, the three sort modes, an unguessed failed row, a whole-batch failure that throws, preset expansion with the shared context prefix, 11 rejected argument shapes that spend no call, and the concurrency cap).

What's new in 0.8.4

The panel stops implying a quota that does not exist, the guard's own Jev calls finally reach the ledger, and the plugin now participates on its own.

• No fabricated quota. TypeSafe is a metered API with no balance endpoint. With

no local budget configured, the pill and the status page no longer print a "额度正常" verdict, no longer draw percentage bars against "∞", and no longer show "额度重置 13 小时 41 分后" — a countdown to a budget that was never set. Without a budget the panel states one thing: pay-as-you-go, nothing resets, only locally measured usage is reported. A "本地预算重置" countdown appears only when a local limit really exists, and it is labelled local.

• Money first. The collapsed pill reads Jev · 今日 $0.000534 · 29 次, and the

popover leads with today's cost and cumulative cost; call counts are secondary rows. Cost is what a pay-as-you-go user actually pays.

• The guard's Jev calls are metered. judgeRisky/judgeStall used to answer the

guard and vanish: a session could report safety.jevCalls: 2 in jev_guard_status while the usage ledger still read "今日调用 0 次". Both verdicts now report latency and token usage through an onJevCall event and are recorded as kind: "judge" ledger entries (tool: "auto-guard").

• Auto-triage (`autoTriage`, on by default). On the first step of every turn one

small Jev call (4 routing questions, ~1.5K input tokens, bounded by autoTriage.timeoutMs) classifies the request — intent, whether an atomic judgment is really needed, which kind, and which Jev tool serves it — and the advice is injected back as a plugin-sourced user message. It runs at most once per (agent, turn), never blocks a step, and fails open on an unknown host shape, a missing key, a timeout or any API error. Being metered, it is visible in the panel, so a working plugin no longer shows zero calls.

• Harder guidance. The system-prompt section now carries mandatory-trigger

rules (classification/routing, priority/severity/satisfaction scoring, truth and compliance checks, structured extraction, choosing among 2-10 options), forbids bypassing Jev "because I already know the answer", and states the real cost magnitude (~$0.00002-0.0001 per call; 25 questions cost the same as 1).

• Tests: new test/triage.test.mjs plus extended assertions in

test/usage.test.mjs, test/client.test.mjs and test/guard.test.mjs; npm test is 30/30. The client half takes effect on a page refresh; the server half (guard metering, auto-triage, guidance, status page) needs a host restart.

What's new in 0.8.3

Settings you save now actually take effect.

• Root cause. The card handed dotted field names to scope.set(field), and that

call stores path: [field] — one segment. A field such as dashboard.basePath was therefore written into the settings document as a single literal key named dashboard.basePath, which no schema ever resolves. Saving reported success and changed nothing: on a live host ~/.dsh/settings.yaml held the flat keys autoGuard.maxJevCallsPerSession: 60 and dashboard.basePath: jev while the resolved document still showed the schema defaults (50 and /jev). Found by reading the live document over the settings/describe RPC after the 0.8.2 restart.

• Fix. The card builds { op: "set", path: [, …] } operations and commits

them in one atomic scope.mutate(ops) call; it only falls back to chained scope.set on hosts that expose no mutate.

• Repair on open. A one-time mount effect rewrites documents written by <= 0.8.2 that

contain literal dotted keys: for each one, set the nested path and unset the literal key, in one atomic mutation, guarded by a WeakSet so it runs once per scope and wrapped so a failed repair can never break the card.

• Client-only: refresh the page — no host restart. test/client.test.mjs PASS 7

covers path splitting, the generated operations, the atomic repair and its idempotence. npm test is 29/29.

What's new in 0.8.2

Two things the board got wrong, plus a usage surface that does not need the board at all.

• `/jev` never 404s again. dashboard.basePath used to be taken verbatim, so a

value such as jev (no leading slash) registered the literal path jev, which no browser request ever matches. Paths are now normalized (leading slash added, trailing slashes trimmed, empty or / falls back to /jev) and both /jev and /jev/ are registered. With dashboard.enabled: false the path still answers with a live status page — today's calls, cost, this instance, today's tokens and latency, quota state — that points at the composer pill and says how to turn the full board on, instead of a bare host 404.

• `/jev/api/usage` — a small read-only JSON route that always mounts (it does not

wait for dashboard.enabled) and projects the same locally measured snapshot the jev_usage tool returns.

• A composer usage pill (conversation.input.right, beside the submit button),

modeled on dsh-opencode-go: Jev · 今日 29 次 · $0.000534 with a warn/danger tone dot, expanding into a 270 px popover — today's calls and cost against their limits, this instance's calls, today's tokens, median and p95 latency, guard denials, the top three tools, time to reset, last refresh and a manual refresh. It polls every 60 s and on tab focus, falls back to the full board API when the pill route is absent, and degrades to Jev · 额度不可用 carrying the exact error — it never throws. It reports local measurements only, never a vendor balance; jev_usage in the conversation stays the full panel.

• Regression tests: test/dashboard.test.mjs covers the path normalizer, the route table

with the board on, off and on a custom basePath, the 200 status page and a failing snapshot; test/client.test.mjs covers the pill payload's three shapes, the label and tone states, and the rendered popover text. npm test is 29/29.

What's new in 0.8.1

Fixes the Settings > Plugins card that showed 未配置 / unconfigured and blank fields while a key *was* configured.

Root cause: the schema registered with the settings service marked the GUI fields with the volatile marker. On a host whose schemastery understands it (3.18.4), resolve() replaces every marked field with a live reference object, and the settings service publishes registration.resolved verbatim — so the wire value was {"enabled":{},"apiKey":{},"apiKeyEnv":{},"quota":{"enabled":{},…}}. The GUI ships its own older schemastery (3.18.2), which cannot decode that ($.enabled expected boolean but got [object Object]), so the card silently fell back to an empty draft: badge "未配置", every field blank. The plugin's own reads were unaffected (it unwraps those references), which is why judgments, the guard and jev_usage kept working throughout.

• One field definition, two schemas. buildConfig() builds both Config

(entry form; keeps the marker for hosts that understand it) and SettingsConfig (plain — the one registered with the settings service), so the 23 fields cannot drift apart.

• The card no longer trusts a single decode path. When the decoded value is

empty it reads the raw base/user layers the service already sends, says so on the card ("服务端配置值未能解码…"), and stays editable and savable.

• A credential reference is not "configured". The badge has three honest

states: literal key / 环境变量 (unverified) / none.

• Regression tests: test/config.test.mjs asserts the registered schema is plain

and JSON-safe on the wire; test/client.test.mjs renders the card against an undecodable snapshot and asserts the badge names the configured reference and never says 未配置. npm test is 29/29.

Reproduced on the wire (deployed plugin, dsh 0.1.5-rc.2, 2026-10-06):

schema · empty-object fields on the wire · GUI decode Config (entry-form only) · 8 + every quota.* · FAILS — $.enabled expected boolean but got [object Object] SettingsConfig (registered) · none · passes; apiKeyEnv="TYPESAFE_API_KEY", enabled=true, autoGuard.denyThreshold=0.8

What's new in 0.8.0

The usage/quota panel: Jev usage is now measured, stored and surfaced — as a tool view, in the overview card, and on the standalone board.

• `jev_usage` — a usage/quota panel you can call. Every decision, choice,

verification and guard event is recorded locally with its measured input tokens, cost, latency, question-type mix and tool name. The tool returns rolling windows (today / 7 days / 30 days / all / session), a daily call and cost series, a projected daily burn rate ("at this pace the daily limit is reached in N h"), and an optional hard stop.

• Local budgets, not invented provider balances. TypeSafe publishes no

balance or quota endpoint (GET /v1/usage, /v1/quota, /v1/account, /v1/balance, /v1/credits, /v1/limits, /v1/billing … all 404; only /v1/models answers). So the panel reports only what this machine actually measured, against limits you set yourself: quota.dailyCallLimit, quota.dailyCostLimitUsd, quota.sessionCallLimit, plus warnAtPercent. The boundary is printed in both the text result and the card — never a fabricated balance.

• Optional hard stop (`quota.enforce`). With enabled + enforce, a call

that would cross a configured limit is refused *before* the API request, with a message naming the budget. Display-only by default.

• Persistence (on by default since 0.8.5; opt-in in 0.8.0–0.8.4). 0.8.0–0.8.4 kept

everything in memory (400 samples) unless you set quota.persist: true; since 0.8.5 up to quota.historyDays (default 30) of daily history is written to $DSH_HOME/jev-usage.json by default (atomic tmp+rename, fail-open), so the cumulative total survives a restart.

• Surfaced everywhere. jev_overview gained a usage block, the standalone

board (dashboard.enabled) gained a quota section with bars, projections and a 30-day sparkline, the settings card gained a "usage/quota" group, and jev_verify now records its 27 calls, tokens and cost instead of only the accuracy string.

• Tests: 28 passing — the new test/usage.test.mjs covers the empty state,

token accounting, batch calls, quota status and the hard stop, persistence across a day boundary, reload, reset and every formatter.

What's new in 0.7.5

The audited release: a three-way review (server, client, packaging) turned up a batch of correctness and honesty bugs, all fixed and pinned by tests — and jev_verify now runs its 27 cases concurrently.

Server

• The session ledger records even when the dashboard is off. record() used

to return early unless the standalone page was mounted, so jev_overview reported 0 calls / 0 tokens / $0 forever while jev_guard_status counted the same events. The in-memory ring now always records; only the HTTP page and the $DSH_HOME/jev-roll.jsonl append stay behind dashboard.enabled (this README and the setting text now say exactly that).

• The guard reports the events it already knew about. The Jev denial path now

emits onSafetyDeny (with confidence and the tool name) and the loop path emits onLoopAdvisory, so the dashboard advisory counter and the Jev-verdict branch of the ledger are no longer dead code.