jasen215/dsh-continual-harness

dsh-continual-harness

English | 中文

A DeepSeek Harness (DSH) plugin for self-improving AI agents, providing continual learning through persistent memory, periodic review and refinement, cross-session knowledge sharing, and automatic rollback on failure. It forms a closed loop of plan → validate → apply → rollback.

The design is inspired by the open-source prime-agent from Prime Intellect, a self-improving coding harness.

Capabilities

A single npm package (dsh-continual-harness) takes effect through the following extension points once mounted:

Architecture

src/
  domain.ts      event declaration merging (SessionEventMap / MessageSourceMap / cordis Events)
  types.ts       HarnessState / RefinementProposal / RefinementResult and other types
  storage.ts     disk read/write of state and history (atomic writes, corruption degradation, local/global merge, jsonl history)
  refine.ts      validation, application, rollback (baseline conflict detection, version increments, growth limit)
  skills.ts      SKILL.md rendering + file reconciliation (generated skills are real dsh skills)
  render.ts      model-facing overview / summary / history rendering (ranked injection)
  usage.ts       injection telemetry keys and in-memory usage aggregation
  wrapup.ts      deterministic session wrap-up suggestions (keep/promote/archive)
  planner.ts     LLM planning prompts and JSON parsing (plan / auto-refine review prompts)
  store.ts       HarnessStore: combined storage + event publishing (session events + agent-scoped events)
  complete.ts    completeViaAgent: completion through ctx.get('llm')
  benchmark.ts   benchmark cases/snapshots + atomic benchmark store persistence
  evaluate.ts    isolated per-cell executor/reviewer evaluation (evidence + score)
  score.ts       code-owned aggregation and ACCEPTED/REJECTED decisions
  tool.ts        harness_refine / harness_wrapup / harness_benchmark tools
  projection.ts  pre-step projection (digest dedup, <harness_state> injection)
  driver.ts      automatic refinement driver (turn-interval gate / compaction gate / cooldown / re-entry guard)
  invariant.ts   runtime invariant plugin
  index.ts       plugin entry and Config
tests/           23 test files, 287 cases (storage / store / refine / rules / planner / driver / approval / audit / logfile / skills / invariant / plugin integration / rank / projection / archive / usage / wrapup / benchmark / evaluate / score / isolation / tool / benchmark integration)

Data layout

<harnessRoot>/                      shared ESP experience root; defaults to ~/.dsh/harness/
  harness_state.json                cross-session global state (ESP)
  refinements.jsonl                 global refinement history (append-only, ESP)
  reviews.jsonl                     cross-batch gate/audit history (ESP extension)
  continual-harness.log             continual-harness implementation log (JSONL, 0600)
  continual-harness.log.1           rotated continual-harness log
  usage.events.jsonl                append-only injection telemetry (lazily loaded into memory on first access)
  benchmark/                        explicit benchmark store (validation layer)
    cases.json                      fixed benchmark cases (draft/frozen + frozen material hashes)
    snapshots/<snapshotId>.json     captured reference snapshots (read-only merged harness state)
    runs.jsonl                      append-only A/B run records (cells + evidence + code-owned decision)
  sessions/<sessionKey>/
    harness_state.json              session-local state (shadows same-id global entries)
    refinements.jsonl               session refinement history

• Skills are real dsh skills: applied skill edits materialize as <name>/SKILL.md bundles (with provenance metadata) under Config.skillsDir, kept in sync by deletes/rollbacks without touching user-owned skills in the same directory.

Experience Solidification Protocol (ESP)

The Experience Solidification Protocol (ESP) is the protocol surface of this capability set, decoupled from this package's implementation:

Any dsh plugin can read and write experience through this protocol (write state files, append history, publish events, inject messages); this package is the protocol's reference implementation and primary consumer (planning / refinement / projection / automatic gate).

Mounting (dsh profile)

Install into a profile in one line (published to npm):

dsh plugin --profile <name> add dsh-continual-harness

The package declares dsh.bundle, so dsh plugin installs it as a profile layer and applies its cordis.patch.yml. Update with dsh plugin --profile <name> update dsh-continual-harness@latest.

Manual overlay (before publish, or to pin a local checkout): apply cordis.patch.yml onto the profile, e.g. ~/.dsh/profiles/<name>/cordis.patch.yml; a patch layer must be a top-level YAML array (insert rows append plugin entries; id-targeted rows override an existing row):

- insert:
    - id: continual-harness
      name: dsh-continual-harness
      config:
        defaultGlobal: true

An id-targeted row replaces the target's whole config instead of merging it, so a one-key override must restate the keys it relies on; anything omitted falls back to the defaults below. Overriding one unrelated key never disables the plugin.

Prerequisites: the tools, agents, session, llm, systemPrompt capability plugins must load before this plugin (its inject declaration enforces that; mounting is deferred until they load).

dsh version compatibility

Verified against dsh 0.2.0-rc.2 — typecheck, lint, build, the full 651-test suite, and a real profile boot on the 0.2.0-rc.2 runtime — and against 0.1.7-rc.1 with the same suite. One peer range admits every runtime from 0.1.5-rc.1 through 0.2.0-rc.2 (>=0.1.5-rc.1 <0.2.1), because that range is exactly what dsh evaluates in its plugin/runtime compatibility gate before mounting a plugin or a profile bundle. dsh 0.1.7 introduced that gate: when a plugin's @deepseek-ai/dsh* peers do not satisfy the running version, its profile bundle is skipped, so widening the range is what keeps one source loadable on both the older and the current runtime. Since dsh 0.1.2-alpha.3 no longer provides @deepseek-ai/dsh-home-paths inside the profile bundle, the plugin declares it as a hard dependency; @deepseek-ai/dsh-invariants is used for types only (dev-time) and is not required at runtime.

Config

Refining

Two entry points: the harness_refine tool (LLM-callable) and the /refine slash command (when the host provides a commands capability).

harness_refine — mode: 'plan' (default) plans from instructions and commits atomically; mode: 'rollback' takes a rollbackId plus an explicit --local / --global scope to revert a committed refinement. Global writes require human approval when requireGlobalApproval is true.

/refine — same semantics, human-typed:

/refine --local organize my memories
/refine --global <instructions>
/refine rollback <id> --local
/refine rollback <id> --global

Bare /refine plans with no instructions in the default scope. Output: status, scope, refinement, applied, rejected, summary, plus a diagnostics: line when enabled.

Governance

Every write path funnels through three guardrails: impact minimization (fixed contract validation; update/delete require a one-line reason; maxEntryGrowth caps per-commit growth), legality hard rejects (base_system_prompt and protected entries are immutable; global entries are read-only during a local refinement), and a necessity soft gate (a declined review never reaches the store). Every committed refinement rolls back by id.

Global writes are zero-approval by default; set requireGlobalApproval: true to ask the user first. Watch the plugin log live with:

tail -f ~/.dsh/harness/continual-harness.log

Benchmark

The validation layer is explicit and single-entry: one harness_benchmark action tool drives the whole workflow and never auto-triggers a refinement — nothing in the benchmark path starts a harness_refine or the automatic gate, and a REJECTED decision is reported and recorded only, never rolled back. The store lives under <harnessRoot>/benchmark/ (see the data layout above).

The minimal sequence is new → add-case → freeze → capture-reference → apply refinement → run → status (frozen case material is immutable and hashed; status lists cases, snapshots, and recent runs). Two steps carry real subtleties:

• capture-reference must run BEFORE the refinement you want to validate: the candidate is later derived as the captured reference plus exactly that refinement, so capturing after the change would make the delta unprovable.
• run evaluates the named refinement A/B against the reference (reference_snapshot_id + refinement_id). The candidate must be the single specified delta — derived from the reference plus the refinement's recorded applied edits and proved in code before any evaluation; a drifted or multi-change candidate is refused (benchmark:run:candidate-delta). Both sides run the same frozen cases in stored order with the same runs/provider/model.

A run returns the code-owned decision (src/score.ts), not a model verdict:

{
  "action": "run",
  "ok": true,
  "run_id": "run-...",
  "refinement_id": "refine-1",
  "status": "ACCEPTED",
  "reference_overall": 70,
  "candidate_overall": 90,
  "regression_cases": [],
  "failed_cells": 0,
  "feedback": ["reference ok", "candidate better"],
  "auto_rollback": false,
  "runs": 1,
  "cells": 2
}

• Scores are 0..100 per cell; a failed cell carries score: null — failure is never counted as 0 — and is excluded from the overall means.
• passThreshold (default 60) is report-only: it never gates acceptance. A run is ACCEPTED only when neither side lacks usable cells, candidate failed cells stay within maxFailedCells, and no overall or per-case regression exceeds regressionTolerance (default 0).
• Every run appends its full record (cells with executor evidence + the decision) to benchmark/runs.jsonl; evaluation reads only the captured snapshots and writes only that record, never touching reviews.jsonl, the harness state, injection telemetry, or skill files.

Development

The plugin is self-contained: devDependencies pin the published @deepseek-ai/* packages (rc versions), so pnpm install, pnpm run typecheck, pnpm test, and pnpm run build (tsc emits lib/types/*.js + *.d.ts; the "." and "./invariant" exports point at the artifacts) all work in a clean checkout — CI and the OIDC release workflow run the same steps. peerDependencies declare the semver ranges consumers (host dsh installations) must satisfy.

scripts/githooks/pre-commit runs the local regression gate (npm run verify, ~4s) on every commit, so a regression CI would catch on push is caught before the commit instead; it bootstraps verify/latest.json when that file does not exist yet. Hooks under .git/ are not versioned, so enable it once per clone:

git config core.hooksPath scripts/githooks

Plugin builds up to 0.3.0 logged the injected overview under a plugin-defined harness-state message source. The released Session format migrations only classify platform source kinds, so one such message makes the whole stored artifact unreadable (cannot safely transform unclassified message source) once a host reads it with a v3-capable dsh. This build logs a classified plugin source instead; stored logs of any generation are repaired offline with node scripts/repair-harness-state-logs.mjs (dry run by default; --apply backs each artifact up and replaces it atomically, and artifacts written within --min-age-seconds are skipped — see --help).

Known Limitations and Deferred Work

• No end-to-end tests with a real LLM: completeViaAgent depends on the loaded llm capability and provider/model configuration; tests cover the planning/review paths with a stub Complete. Real e2e requires DEEPSEEK_API_KEY.
• compaction/end is not part of the plugin's type union; the driver triggers it via string comparison after type narrowing, and the gate is silently skipped when the compaction capability is not loaded.
• Projection dedup is an in-process WeakMap<Agent, digest>: the first step after a session restart re-injects (stateless and idempotent, but one extra injection).
• Concurrent writes are last-writer-wins: multiple processes refining the same directory concurrently may overwrite each other; baseline conflict detection during planning can only catch read-after-write races, not serialize them.
• A failed automatic refinement degrades silently (only logged) and never interrupts the session.
• A content-shrink guard (rejecting updates that shrink an entry too far in one commit) is a planned follow-up and is not yet implemented; today only maxEntryGrowth caps how much an update may grow an entry.
• A dedicated governance tool entry is deferred.