dsh-goalloop-agentteams

dsh-goalloop-agentteams

goal-loop as a DeepSeek Harness plugin — the judging face of the goal-loop skill, wired into native Agent Teams, the task board, and DSH's completion-interception pipeline. The skill itself ships as two runtime skills (goal-loop, goal-loop-contract) so the model knows the protocol; the plugin makes the protocol enforceable.

Contract-first acceptance criteria, a gate that re-runs every check itself (zero model trust), a human approval stamp that freezes done, verdicts bound to a content digest, and a hard deny on false completion at update_goal / team-task / task-board surfaces via tools/pre-execute. The portable team-file protocol and dual-ledger bookkeeping are deliberately not ported — native Agent Teams is strictly stronger.

v0.10.0 — judgment integrity hardening. Highlights since 0.6.2:

• Trust the ledger, or stop. state.json is HMAC-signed with a key kept outside the workspace; a forged, edited, truncated or deleted ledger is state-error (exit 4), never silently rebuilt. The watermark/BLOCKED latch/approval can no longer be edited away by the agent under evaluation.
• The lock protects the critical section, not the whole tool surface. Read-only tools and ordinary work never take it; it is acquired only for completion claims, carries a pid + per-acquisition token, releases only if it still owns the lock, reclaims immediately when the holder process is dead, and gate.mjs unlock is the human recovery path.
• No free PASS. The first measurement of a maximize/minimize series only records the watermark (verdict unverifiable); the metric identity is the AC id plus a hash of the statement, so editing the instrument keeps the history while re-targeting starts a new series.
• The gate runs the judge itself. With calib: the probe must contain a {file} placeholder; the gate runs it against the known-good fixture (must exit 0) and the known-bad one (must exit non-zero). A score: counts only when the judge prints it, and evidence must resolve inside the evaluated workspace.
• Honest exit codes. check exits 0 GO / 2 NO-GO / 3 BLOCKED / 4 state-error / 5 busy; a gate that never ran can no longer be mistaken for a failed conclusion.
• Fail closed on a missing workspace. No cwd means no judgment (previously it fell back to process.cwd(), where no contract exists, so every claim passed). goal_gate_stamp now goes through the approval seam by default.
• BROKEN is decisive. Missing instrument (rc 127), failed probe, metric miss, unmatched sd:/var:, unbound/stale judged evidence, missing judge and calibration defects are broken → NO-GO; they can no longer hide inside the “≤1/3 unverifiable” allowance.
• R3 approval stamp. goal_gate_stamp / /goal-stamp freezes the contract and budgets; any later edit makes completion claims fail as contract-revised until an explicit re-stamp.
• Content-bound digests. Same-size edits change the digest; a check that mutates the workspace is refused as check-mutated-tree; build-output directories are no longer silently excluded.
• Objective surface. minimize, !=, data-driven sd:/var: noise bands, first-run watermark only (no free PASS), and per-round efficiency with strategy attribution.
• Budgets are fuses. Round and wall-clock budgets bind at stamp and deny non-GO completion claims; goal-reset --new-loop is the human recovery path.
• DSH-native. task_board_manage move-done is intercepted; dispatch:true and goal_gate_dispatch create real native Agent Teams tasks for a Team Lead; read-only goal_gate_status / goal_gate_audit.
• Complex-case fixes (v0.9.1). Claim-time delta no longer requires a second improvement, grill no longer flags ordered scopes / bound judged contracts, and the fallback CLI now takes the ledger lock; examples/plugin-e2e covers all three.
• Host-contract fix (v0.7.1). Tool parameters always compile to an object-rooted provider schema; the test-only dsh-tools shim is never left inside the link-installed package (that shim caused schema must be a JSON Schema of 'type: "object"', got 'type: null' at boot).
• Team staffing (v0.9). goal_gate_staff turns ACs (or dependency waves) into teammate specs, previews by default, and with spawn:true spawns real native teammates, creates/assigns their board tasks and sends an AC brief. goal_gate_collect maps completed/in-progress/missing tasks back to ACs (readyToCheck); goal_gate_judge creates fresh-reviewer tasks for judged ACs with digest/calib requirements.
• Agent Teams-native dispatch (v0.8). ACs may declare scope: and depends:; dispatch creates dependency-ordered waves with blockedBy/writeScopes, reuses pending tasks instead of duplicating them, and can assign owners. goal_gate_status/goal_gate_check return a team snapshot and nextActions; a ledger lock serializes concurrent member operations.
• Grilling eval design (v0.8). goal-loop-grill skill + goal_gate_grill tool adversarially audit the contract: tautologies, unfalsifiable claims, Goodhart proxies, missing runtime evidence, unbound judged verdicts, missing noise bands, overlapping scopes. goal_gate_baseline measures optimize-AC mean/spread before stamping.
• Packaged skills + commands. Three skills appear in the session skill catalog; /goal-loop-at, /goal-gate, /goal-stamp, /goal-audit, /goal-reset, /goal-grill, /goal-baseline are registered.

中文简介

dsh-goalloop-agentteams 把 goal-loop skill 做成 DeepSeek Harness 插件：判命面由确定性门控执行（模型永不自宣完成），闭环面接入原生 Agent Teams / 任务板，知识面是运行时注册的两个 skill。

• 门控：自己重跑每条 AC 的 check（零模型参与）；broken（仪器坏了、判官缺失、证据失联、取数失败）是决定性 NO-GO；只有门控能发 GO。
• R3 盖章：goal_gate_stamp / /goal-stamp 冻结契约与预算；盖章后改契约，完成声明一律 contract-revised，重新批准才放行。
• 证据绑定：judged 评审必须含 artifact: + digest: sha256:，且证据文件必须落在被评估的工作区内；calib: 由门控自己拿 probe 对已知好/坏夹具各跑一次（好夹具必须过、坏夹具必须不过），不采信评审自述的 calib-good: PASS；检查自己改工作区（check-mutated-tree）判无效——测量产物请写 .goal-gate/evidence/。
• 闭环：/goal-loop-at <目标> 建契约骨架 + 轮次配置 + 协议 + 建议任务；score 不回退、轮次/墙钟预算到点停、假完成 2 次 BLOCKED（人类 /goal-reset 才解）。
• 分级治理：update_goal(complete) 永远硬拦；任务级完成声明按 exit: strict|goal-only 分级；task_board_manage move-done 同样受管——多任务循环不会死锁，也不会绕过门控。

完整中文文档见 README.zh.md。

The design rationale (right agentic loop, right eval, right metric, and how each maps onto Agent Teams) is in docs/agent-teams-loop.md.

Why / 为什么需要它

DSH already has two layers of completion judgement — but neither is a deterministic gate:

dsh-goalloop-agentteams is the hard floor underneath both: it re-runs every AC's check command itself, binds verdicts to a content digest, and counts caught false-completes — two catches → BLOCKED. Deterministic gate = floor, LLM judge = semantic layer, quality kinds = step contract. Stacked, they close the loop.

Install / 安装

dsh plugin --profile <profile> add dsh-goalloop-agentteams

Or from a checkout / GitHub:

dsh plugin --profile <profile> add /path/to/dsh-goalloop-agentteams
dsh plugin --profile <profile> add github:samirliu/dsh-goalloop-agentteams

The package declares dsh.bundle with a cordis.patch.yml beside it, so dsh plugin add mounts the goal-gate row directly. Restart the profile after install (host code, commands and skills are composed at boot).

⚠️ Release pinning: install from a checkout or a release ≥ v0.7.0. The v0.6.x releases predate the DSH 0.2.1-rc.2 tool-registration contract (schema-only output with no render), so they install cleanly and then fail on every successful tool call — --dump-config cannot tell them apart.

⚠️ Entry contract: the entry must export name equal to the cordis.patch.yml id (goal-gate) alongside inject and apply. A missing name makes the Host silently skip the bundle. link: installs additionally may resolve @deepseek-ai/dsh-tools differently than registry installs; the entry tries the bare package first and falls back to a same-shape shim, and both paths are exercised by tests.

Platforms / 平台支持

The gate re-runs every AC check itself, so it owns the shell — the choice is explicit and reported, never a silent platform assumption:

• The shell that judged a round is reported as shell in goal_gate_check, goal_gate_status and bin/gate.mjs check output.
• A missing shell binary is broken with an explicit "gate shell not found …" reason (config problem, not a failing artifact) — never a mute failed.
• preflight emits posix-syntax-on-cmd when a check looks POSIX (test, grep, &&, $VAR) while the Windows gate would run cmd.exe.
• On Windows either write checks in cmd/pwsh syntax, or point the plugin at a POSIX shell:

- id: goal-gate
  name: dsh-goalloop-agentteams
  config:
    shell: 'C:\Program Files\Git\usr\bin\sh.exe'

Precedence: bin/gate.mjs --shell / plugin config.shell > GOAL_GATE_SHELL > platform default. A POSIX-style contract on a cmd.exe gate is a trap the preflight warning and the reported shell field are designed to expose before two false completes latch BLOCKED.

Plugin config / 插件配置

cordis.patch.yml mounts the row with config: {}; add any of these keys to tune it:

Quick start / 快速开始

/goal-loop-at ship csv2jsonl with hard row validation and docs

or let the model call goal_loop_at({objective, acs?, maxRounds?, dispatch?}). Then run the four-step loop:

1. Contract. Load the goal-loop-contract skill when criteria are fuzzy. Each AC is one line: a yes/no statement plus a check that can actually FAIL. At least one AC must execute the deliverable itself at runtime. Present the one-screen summary.
2. Approve + stamp. The user approves; the model calls goal_gate_stamp({approvedBy, maxRounds?, timeBudgetMinutes?}), or the user types /goal-stamp. The stamp freezes the contract; editing it later voids the approval.
3. Dispatch. Native Agent Teams (spawn_teammate / team_task_create) first, task-board cards (task_board_create) when teams are unavailable, solo otherwise. goal_loop_at({dispatch:true}) creates one team task per AC for a Team Lead; goal_gate_dispatch turns current failures into repair tasks. Disjoint write scopes; freeze interfaces first.
4. Join + gate. goal_gate_check({strategy?, note?}) is the only completion authority. NO-GO → repair from failedActions, check again. GO → goal_gate_audit → deliver. Two caught false goal-completes → BLOCKED; only a human /goal-reset clears it.

Tool surface / 工具面

Commands: /goal-loop-at <objective> [--no-grill], /goal-gate (status), /goal-stamp [--restamp] [--max-rounds=N] [--time-budget-minutes=N] [approvedBy], /goal-audit, /goal-grill, /goal-baseline [samples], /goal-collect, /goal-judge [reviewer], /goal-doctor, /goal-reset [--new-loop] [reason].

Runtime skills: goal-loop (the playbook), goal-loop-contract (contract forge / stamp checklist) and goal-loop-grill (adversarial eval/metric design). They are registered with ctx.skills.register and appear in the model's skill catalog; no filesystem install is needed.

Contract grammar / 契约文法

objective: <one-line goal>
deliverable: <path to the artifact itself>
exit: goal-only | strict
AC-N | <yes/no statement> | check: `<failable command>` | [probe: `<instrument check>`] | [metric: `<regex, one capture group>`] | [baseline: delta|abs] | [timeout: <seconds>] | [tolerance: <relative>] | [sd: `<regex>`] | [var: `<regex>`] | [calib: good=<file> bad=<file>] | [scope: <path[,path...]>] | [depends: AC-x[,AC-y]] | expected: <spec>

Specs: exit=0 | exit=N | <=N | >=N | <N | >N | =N | !=N | maximize | minimize | judged.

• One AC per line, fixed token order. A line that starts with AC-N but does not parse makes the gate refuse the whole contract (state-error); a leading - AC-N list marker is accepted. Duplicate AC ids, unknown dependencies and dependency cycles are refused.
• scope: is the workspace-relative write scope used by team dispatch (writeScopes); depends: builds the dependency graph (blockedBy, plan.waves).
• probe: runs first for numeric/optimization ACs and must pass; for judged it is the judge (exit code = verdict). A failed probe is broken, never passed.
• metric: extracts one capture group from stdout (ANSI stripped); without it the last number is used. Missing required metric = broken.
• maximize / minimize compare against the previous accepted value; the first run only records the water mark (verdict unverifiable, no pass — a fresh metric series must not hand out a free PASS). baseline: delta demands strict improvement, abs (default) forbids regression. tolerance: is a relative noise band; sd:/var: widen it to max(2*spread/|prev|, tolerance) (var: is square-rooted); declared-but-unmatched spread is broken. The metric identity is the AC id plus a hash of the statement, so editing the instrument keeps the history while re-targeting the statement starts a new series.
• judged requires the verdict to carry artifact: <path> and digest: sha256:<64hex> (recomputed by the gate). Unbound, stale or unreadable evidence is broken, and the artifact must resolve inside the evaluated workspace. With calib: good=<f> bad=<f> the gate runs the probe (which must contain a {file} placeholder) against both fixtures itself: the known-good one must exit 0 and the known-bad one must exit non-zero. Self-reported calib-good: PASS / calib-bad: FAIL lines are not trusted.
• Evidence produced by checks (renders, logs, judge inputs) belongs under .goal-gate/evidence/ — it is excluded from the digest. Checks that write anywhere else in the workspace fail the round as check-mutated-tree.
• deliverable: is advisory in the gate but drives preflight: a contract with no check touching the deliverable and no runtime AC gets deliverable-unverified / no-runtime-ac warnings before a round is burned.

Gate semantics / 门控语义

goal_gate_check returns rc=0 GO, rc=2 NO-GO, rc=3 BLOCKED, rc=4 state error. Every round is appended to .goal-gate/history.jsonl.

Additional outputs: score (passed/total), bestScore (high-water), regression (score dropped), trend (last five), efficiency (scoreDelta, slope, per-strategy attribution), digestBefore/digestAfter (a check that moved the tree returns check-mutated-tree), timeRemainingMinutes, timeExhausted.

Approval, budgets and blockers / 批准、预算与阻塞

• R3 stamp. goal_gate_stamp({approvedBy, maxRounds?, timeBudgetMinutes?, restamp?, note?}) writes the approval to state.json and may bind the budgets. A contract edit changes its stamp; completion claims are denied contract-revised until restamp:true (or /goal-stamp).
• requireStamp: true in plugin config makes an approved stamp mandatory before any completion claim (default false; the stamp remains recommended and is what /goal-loop-at's protocol asks for).
• requireHumanApproval (default true) makes goal_gate_stamp itself go through the DSH approval seam ({kind:'ask'} in tools/pre-execute): stamping freezes what counts as done, so it must be the human's decision. Under an approval policy of never this fails closed; set requireHumanApproval: false explicitly for headless loops.
• Round budget (maxRounds, default 8) and wall-clock budget (timeBudgetMinutes) are fuses: a non-GO completion claim at/over budget is denied (rounds-exhausted / time-budget). Green delivery is never confiscated at the time fuse.
• False-complete ratchet: a denied update_goal(complete) means the artifact is not done. Two denied goal-completes → BLOCKED; all claims freeze. /goal-reset [--new-loop] is the human-only recovery (there is deliberately no model-facing reset tool).
• Loop identity: a second goal_loop_at with a different objective refuses (objective-drift); changing the objective is changing the definition of done, so edit + re-approve or start a fresh workspace/loop.

Interception / 完成声明拦截

Mounted on tools/pre-execute; always returns structured {kind:'deny'} (never throw):

• update_goal(action:'complete') — always hard-gated by the full contract, with the R1 counter.
• Task-level completion — agent_teams_update_task(status:'completed'), team_task_update(action:'complete'), legacy update_task, and task_board_manage(action:'move-done'). Under exit: strict they are gated like a goal claim; under exit: goal-only they are recorded (partialCompletes) and pass, so multi-task team loops do not deadlock on teammates' unfinished ACs. BLOCKED freezes every claim.
• Before running the gate it checks approval freshness (contract-revised) and, when configured, requireStamp (unstamped). A malformed contract is denied as state-error; no contract at all passes through.

Dispatch / 派工

• goal_loop_at({dispatch:true}) creates one native Agent Teams task per AC when the caller is a Team Lead (ctx.agentTeams); other callers get suggestions instead of a fake success.
• goal_gate_dispatch({includeUnverifiable?}) re-runs the checks without recording a round and creates tasks for failed/broken (and optionally unverifiable) ACs. Each task description carries the exact command and spec.
• Without a Teams service, use task_board_create / solo work. The plugin never writes the task-board ledger directly.

Agent Teams loop / Agent Teams 闭环

The plugin adapts to the native team service opportunistically (ctx.get('agentTeams')); it never requires it and never writes the task-board ledger itself.

Eval design with grilling / 用 grilling 设计评估

goal_gate_grill is a read-only attack on the contract, derived from goal-loop's grilling/contract-forge discipline. It returns findings[severity, ac, issue, suggestion], the remaining questions for the user, metricDesign advice per optimize AC, and goodhart gaming paths. verdict: ready requires no high/medium findings. The goal-loop-grill skill teaches the interview frontier: ask what "done" means