PerryLink/dsh-auto-review

🤖 dsh-auto-review

Second-model AI approval for DeepSeek Harness — a read-only reviewer subagent decides allow/deny on the approval chain, fail-closed by default.

*When an action crosses the sandbox boundary, a second model reads the evidence and returns a verdict with a reason — so humans approve nothing while nothing unsafe slips through.*

Official repository. This is the only official repository of dsh-auto-review, maintained by PerryLink. Same-name repositories under other accounts are not affiliated.

English · 简体中文 · Español · Português · हिन्दी

dsh plugin --profile <name> add dsh-auto-review

📖 Ecosystem knowledge base — measured data, not marketing: plugin development guide · plugin-selection data · maintenance criteria.

Install & uninstall

• git channel (latest main): dsh plugin --profile web add github:PerryLink/dsh-auto-review — the isolated prepare build needs the single allowBuilds: { esbuild: true } key the dsh CLI prints for dsh-auto-review.
• npm channel (published releases): dsh plugin --profile web add dsh-auto-review.
• 1024 store channel: npm i -g dsh1024 once, then dsh1024 plugin --profile web add dsh-auto-review (counts toward the deepseek1024.com install ranking).
• tarball channel: pnpm pack in this repo, then dsh plugin --profile web add ./dsh-auto-review-.tgz.
• uninstall: dsh plugin --profile web remove dsh-auto-review (or remove the row from the profile patch).
• native build scripts: when dsh plugin add stops at ERR_PNPM_IGNORED_BUILDS for koffi / node-pty (pulled in by the eval harness), run pnpm approve-builds to approve those build scripts.

⭐ 如果它帮到了你

这个插件是 DSH 插件家族的一员（95 个，全部 Apache-2.0）。如果你在用，给个 star —— 它不会解锁任何功能，但会让下一个人在搜索里更容易找到它。

*English:* part of a 95 plugin family for DeepSeek Harness. If it is useful, a star helps the next person find it — nothing is gated behind it.

What is dsh-auto-review?

Second-model AI approval for DeepSeek Harness — a read-only reviewer subagent decides allow/deny on the approval chain, fail-closed by default.

When an action crosses the sandbox boundary, a second model reads the evidence and returns a verdict with a reason — so humans approve nothing while nothing unsafe slips through.

*Workflow overview: request → policy / review → verdict. The examples below use the real command surface.*

Compatibility

Surface · Status Harness · DeepSeek Harness dsh-v0.2.1-rc.2 (verified 2026-10-18): engines.dsh >=0.2.1-0 =24.0.0 Platforms · All (host answerer; optional Web review panel via the session-projection capability) Model · Any (the reviewer inherits the session agent's route; reviewerModel overrides)

What you get

dsh-auto-review puts a second model on the approval/request answerer chain:

1. Official seam — an answerer that claims only the requests it owns (ai policy) and delegates everything else via next(); the human approval flow is never short-circuited.
2. Read-only reviewer subagent — a one-shot fork with a read/glob/grep tool allow-list returns a structured verdict { decision, reason, riskLevel }. Reviewer asks are recognized by identity and delegated; maxDepth + the allow-list keep the reviewer non-delegating.
3. Fail closed — reviewer crash, timeout, or schema mismatch resolves through fallbackPolicy (default rejected); a deny verdict feeds its reason back to the calling model.
4. Config-driven routing — per-tool policies (ai/human/never) plus regex risk rules, all changeable from cordis.yml.
5. Deny reasons reach the model — the reviewer's reason is injected into the denied tool result (callId-linked); fallback and never-policy rejections inject auditable markers too ([auto-review] / [auto-review-fallback] / [auto-review-never]).
6. Full audit trail — log-only autoReview/verdict + autoReview/rejection session events (envelope ignorable: true) plus an optional invariant companion enforcing marker ⟺ event.
7. Safety knobs — a rejection circuit breaker (3 consecutive denials, or 6 of the last 10 verdicts, per turn), a risk-level policy, a one-shot /auto-review approve override, and a never-policy hard disable that explains itself to the model.
8. Optional reviewer context — a bounded compact transcript (contextBudget) plus a Codex-style Markdown ruling policy (reviewerPolicyText).

Every decision reconstructs from the session log: approval/asked → autoReview/verdict (or autoReview/rejection) → approval/decided.

Why a second model instead of rules?

Pattern-based auto-approvers decide before dispatch, with no evidence. dsh-auto-review gives the decision to a reviewer subagent that reads the actual workspace (through its read-only tool face), the already-streamed tool-call arguments (sensitive values redacted), the request reason, and your risk rules — then returns a structured verdict. A deny verdict feeds its reason back to the calling model, so the agent learns why instead of retrying blindly.

Quick start

dsh plugin --profile web add github:PerryLink/dsh-auto-review

# 1. install the bundle into your profile
dsh plugin --profile web add github:PerryLink/dsh-auto-review

# or from npm (published releases)
dsh plugin --profile web add dsh-auto-review

# 2. restart and verify the row
dsh --profile web --dump-config | grep -A4 'id: auto-review'

Out of the box the shipped patch AI-reviews bash and write; every other tool (including edit — in-place modification) delegates to the human chain. Add edit: ai explicitly if you accept in-place edits without a human in the loop.

Examples

These are the plugin's real entry points; install once, then call one.

Detailed example

Command

~~~sh /auto-review status ~~~

Output

~~~text All-time: 1 allows, 0 denies, 1 fallbacks, 1 never rejects (avg 10 ms per verdict). Recent verdicts: write: fallback(timeout), bash: allow ~~~

*Observed from test/command.spec.ts:123-160 (assertions at :158-159)*

Failure branch

Input

~~~text /auto-review approve 2 ~~~

Output

~~~text No recent denial #2 in this session (0 recorded). ~~~

*Observed from plugins/dsh-auto-review/src/messages.ts:53; plugins/dsh-auto-review/src/runtime.ts:1039; plugins/dsh-auto-review/test/command.spec.ts:109-113*

Brief examples

• auto-review
• dsh-eval

Configuration

All tunables are Schemastery Config fields (changeable from cordis.yml). An id-targeted override replaces the whole row — restate every key you need.

Key · Default · Meaning enableByDefault · true · Sessions start with auto-review enabled; /auto-review on\ · off writes a durable override that beats this toolsPolicy.default · human · Policy for unlisted tools (delegate to the human answerer) toolsPolicy.overrides · {} · Per-tool policy: ai / human / never riskRules · [] · {pattern, policy, field?} matched before the tool table; field selects reason (default), toolName, or arguments reviewerProvider · fork · Subagent provider for the reviewer (in-process fork backend) reviewerPreset · *(none)* · Read-only Agent Preset id; required when the host exposes startActivation (0.2.1+) instead of toolFilter reviewerModel · *(inherit)* · Reviewer model id; unset inherits the session agent's route reviewerModelProvider · *(inherit)* · LLM provider route for reviewerModel; unset inherits the session agent's provider reviewerTimeoutMs · 60000 · Verdict deadline; on expiry the fallback policy applies reviewerTools · [read, glob, grep] · The reviewer child's tool allow-list (must be non-empty) fallbackPolicy · rejected · Reviewer failure: rejected (fail closed) / delegate / allow-once maxReviewsPerTurn · 10 · Real AI-verdict budget per open turn; beyond it, requests delegate maxFailuresPerTurn · 10 · Reviewer-failure budget per open turn reasonMaxChars · 2000 · Cap for reviewer reasons and the redacted argument preview reviewerGuidance · *(none)* · Optional advisory guidance appended to the reviewer prompt reviewerPolicyText · *(none)* · Markdown ruling policy injected into the reviewer prompt (Codex-style) denyGuidance · *(anti-circumvention text)* · Guidance appended to every injected deny reason contextBudget · {turns: 2, maxChars: 4000} · Compact transcript budget for the reviewer prompt (the open turn plus the one before it); turns: 0 disables the section — and a blind reviewer denies user-authorized actions, so the runtime warns when 0 meets an ai policy. The character budget is spent on the most recent lines riskPolicy · {maxAutoAllow: high, onHighRisk: delegate} · allow verdicts above maxAutoAllow delegate or deny circuitBreaker · {consecutiveDenies: 3, windowDenies: 6, windowSize: 10, action: delegate} · Rejection circuit breaker overrideTtlMs · 300000 · How long a /auto-review approve override stays usable verdictCacheTtlMs · 60000 · Reuse a recent verdict for an identical tool + arguments fingerprint; 0 disables the cache. Only applies with contextBudget.turns: 0 — a transcript-dependent verdict is not replayable from tool + arguments alone verdictCacheMaxEntries · 256 · Maximum cached fingerprints before oldest-eviction language · en · UI language of the /auto-review command output (en \ · zh) allowUnmarkedAudit · false · Force session-log audit on hosts that drop the ignorable marker or fail-closed on unknown event types (host 0.1.2-rc.1+) (dangerous: unmarked events make sessions unresumable elsewhere); default is detect-and-degrade (adapted 2026-09-02, re-verified against 0.1.5-rc.2 on 2026-09-11): the session envelope keeps its ignorable field for stored-log read compatibility only - Session.append still cannot stamp it, so audit-gate behavior is unchanged.

Example (annotated full form: fixtures/config/config-full.yaml):

- insert:
    - id: auto-review
      name: dsh-auto-review
      config:
        toolsPolicy:
          overrides: { bash: ai, write: ai }
        riskRules:
          - pattern: '([Rr][Mm]\s+(-[a-z]+\s+)*/|[Gg][Ii][Tt]\s+push\s+--force)'
            policy: never
          - pattern: 'write'
            policy: never
            field: toolName
        reviewerTimeoutMs: 30000
        fallbackPolicy: delegate
        riskPolicy: { maxAutoAllow: medium, onHighRisk: delegate }
        circuitBreaker: { consecutiveDenies: 3, windowDenies: 6, windowSize: 10, action: delegate }

Where the config actually comes from

`~/.dsh/settings.yaml` is NOT a config source for this plugin. An auto-review: block there has no effect and produces no warning: like every DSH function plugin, dsh-auto-review receives its Config from the row the loader mounts it with — the profile's cordis patch layer. (Some other DSH plugins additionally read the settings service, so the inconsistency is easy to trip over, and the symptom is indistinguishable from the reviewer simply denying.)

Put the configuration in your profile's cordis.patch.yml. An id-targeted override replaces the whole config row, so restate every key you need — dropping toolsPolicy silently returns bash/write to the schema default human and the reviewer stops running at all:

- id: auto-review
  config:
    toolsPolicy:
      overrides: { bash: ai, write: ai }
    contextBudget: { turns: 4, maxChars: 8000 }

Tools & surfaces

Surface · Kind · Notes auto-review · answerer · approval/request waterfall answerer — claims ai-policy requests, delegates the rest via next() /auto-review · command · on\ · off\ · status\ · approve [n] — durable per-session override, budgets, and cumulative statistics deny-reason injection · listener · tools/post-execute — verdict / fallback / never reasons fed back to the denied tool result autoReview · session projection · Folded from the log-only autoReview/* events Web review panel · client · Session-header action: switch, budgets, statistics, recent verdicts, one-shot approve dsh-eval · CLI · YAML-driven agent evaluation engine (bin/dsh-eval.mjs) invariant companion · invariant · dsh-auto-review/invariant (optional; needs the invariants service)

Session command

/auto-review on|off|status|approve [n]

on/off append the durable autoReview/state override (the fold survives restart/resume — replay IS the state) and inject a switch notice the model sees (logged as a user/message event). status reports the effective state, both per-turn budgets (AI verdicts and reviewer failures), a tripped circuit breaker when one is active, and the session's cumulative statistics (allows/denies/fallbacks/never rejects, mean duration, recent verdicts). approve [n] records a single-use autoReview/override for the n-th most recent denial (1 = most recent): the next same-tool review within overrideTtlMs carries the authorization as reviewer context — the reviewer still decides, and the override is consumed by that review regardless of its outcome.

Web review panel

In the Web GUI (web profile), the package contributes a session-header action (AI Review) that opens a panel with the session's auto-review state: the switch with on/off buttons (they execute /auto-review on|off), both per-turn budgets, cumulative statistics (including hard-disable rejections and cache hits), the circuit trip, the recent verdicts, and one-shot approve buttons for recent denials (they execute /auto-review approve [n]).

How it is wired:

• The host registers an autoReview session projection (folded from the log-only autoReview/* events) and serves it through the session-projection channel.
• The browser half is a client module (auto-discovered from the dsh.client declaration) registered on the conversation.session.header.actions seat.
• No extra patch rows are needed: the panel loads whenever the plugin is installed in a profile whose web build provides the session-projection capability (the web profile does). Without that capability the panel reports itself unavailable; the answerer is unaffected.

The panel reads only whole projection values — it never receives the raw session event stream.

How it works

approval/request waterfall (answerer chain)
                        │
┌───────────────────────┴──────────────────────┐
│ dsh-auto-review answerer                     │
│  · session enabled?  · policy = ai?         │   no ── next() ──▶ human answerer (UI)
│  · risk rules → toolsPolicy → default       │
└───────────────────────┬──────────────────────┘
                        │ yes
                        ▼
        ┌───────────────────────────────────┐
        │ reviewer subagent (fork, one-shot)│
        │  · toolFilter: read/glob/grep     │
        │  · outputSchema: {decision,       │
        │    reason, riskLevel}             │
        │  · timeout + req.signal abort     │
        └───────────────┬───────────────────┘
                        │ verdict / failure (fail-closed fallback)
                        ▼
 allow → allowed-once        deny → rejected + reason injected into the
                                       denied tool result (callId-linked)
                        │   never → rejected + [auto-review-never] feedback
                        │            (hard disable, no reviewer runs)
                        ▼
 audit: approval/asked → autoReview/verdict | autoReview/rejection
        → approval/decided (session events, log-only, invariant-checked)

Composition order. The answerer runs at its registration position in the waterfall: if a human UI answerer is composed BEFORE the auto-review row, humans answer first and the reviewer only sees what is delegated downstream. Verify with dsh --profile --dump-config and place the auto-review row before your human answerer rows when you want ai-policy tools routed to the reviewer first.

dsh-eval — agent evaluation engine

Beyond the approval reviewer, dsh-auto-review ships dsh-eval: a YAML-driven agent evaluation platform that runs real headless DSH sessions (one isolated agent + scratch workspace per case, the official Minimal persona as the baseline system prompt), collects the tool-call trace from the session event log, and evaluates structured assertions plus an optional second-model review — the same reviewer seam as the approval answerer.

# eval/cases/demo.yaml (abridged)
suite:
  name: my-suite
  cases:
    - id: math-output
      input: Solve 17 × 24 and reply with only the final number, nothing else.
      expect:
        output: { contains: "408" }
    - id: glob-trace
      seedFrom: '.'
      input: Use the glob tool with pattern "src/**" to list the source files…
      expect:
        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
        results: [{ tool: glob, contains: "index.ts" }]

Run it (a DeepSeek API key must be in the environment):

dsh-eval eval/cases --model deepseek-flash --timeout-ms 240000 --out .eval-reports

Assertion families

The expect block supports six assertion families; each assertion is evaluated independently and reports its own pass/fail with expected/actual values, so a failing case explains itself without a rerun.

Family · DSL keys · What it gates Tool trace · toolCalls, toolCallsExact, noToolCalls, results · ordered tool-call sequence (subsequence with skips), exact name sequence, per-tool result (isError/contains/regex) Output & budget · output, turnEnds, maxTokens · final-output substring/regex, turn outcome, token budget Prompt regression · prompt · the rendered system prompt must match a committed baseline (or a baselineFrom file); any drift is reported as a side-by-side diff, with allowedChanges regexes to whitelist intended edits Stress metrics · stress · P99 step latency (maxP99Ms), worst time-to-first-token (maxTtftMs), aggregate token generation speed (minTokensPerSecond) Fairness · bias · bias radar over the final output: per-category regex counts (categories), hard forbid patterns, maxHits/maxCategoryHits caps Second-model review · review · a supplementary pass/fail verdict from the reviewer subagent (a separate layer, same seam as the approval reviewer)

- id: regression-gate
  input: Answer in one sentence.
  expect:
    prompt:
      baseline: "You are a helpful software engineer assistant."
      allowedChanges: ["copyright-year"]
    stress:
      maxP99Ms: 8000
      maxTtftMs: 3000
      minTokensPerSecond: 20
    bias:
      categories: { gender: ["[Hh]e is (un)?stable"] }
      forbid: ["[Ss]crew that"]
      maxCategoryHits: 0

CI gate: the process exits 0 only when every case of every suite passed — failing evaluations fail the build. Each case leaves a replayable session JSONL and a trace JSON beside report.md/report.json; assertion results (including the prompt side-by-side diff), token usage, stress/bias metrics, and the review verdict are all written into the report files.

- name: dsh-eval
  run: npx dsh-eval eval/cases --model deepseek-flash --timeout-ms 240000 --out .eval-reports
  env:
    DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}

dsh-eval differs from openai/codex-research: codex-research scores agent trajectories for research comparison; dsh-eval is a declarative pass/fail regression harness — YAML cases, structured trace/prompt/stress/bias assertions, an optional second-model review, and a CI exit code — for gating any DSH agent, not research ranking.

MCP server (standalone)

dsh-auto-review also ships a stdio MCP server (dsh-auto-review-mcp) so external MCP clients (Claude, Codex, …) can consume a deterministic review path without a harness. It speaks JSON-RPC 2.0 over newline-delimited JSON (NDJSON) — one JSON object per line, no Content-Length framing.