dsh-test

@ruihuahe/dsh-test

English | 中文

dsh-test is a dev-only black-box acceptance controller for DeepSeek Harness Web profiles and plugins. It starts a real DSH profile in a run-owned home, drives the real browser UI, verifies external facts, and stores replayable JSON state plus hashed evidence.

It is not a Cordis plugin, an MCP server, or a production dependency. DSH remains the system under test.

What changed in 0.2

• Built-in --dsh-root adapter that launches the public DSH CLI on loopback with a dynamic port.
• Run-owned DSH_HOME; only the selected profile definition is copied, while installed dependencies are reused through links.
• Automatic adapter reconstruction for later workflow, resume, and stop invocations.
• Real Playwright browser loaded from the DSH checkout, with system Chrome and bundled Chromium fallbacks.
• ARIA snapshots, screenshots, normalized console/network events, assertions, sanitized runtime logs, SHA-256 hashes, and renderer-verification gates.
• Durable browser storage and ARIA refs across workflow-process restarts.
• Crash-safe workflow state writes, definition hashes, retry attempts, and non-zero CLI results for failed workflows.
• Public manifests contain only the token-free base URL. The DSH browser token stays in a mode-0600 run state file and is redacted from evidence.

Compatibility is recorded in dsh.lock.json. The current live verification target is DSH 0.1.5-rc.2 at c291e79.

Quick start with DSH

Build once:

pnpm install
pnpm build

Start a real profile. This example uses the local Qualy plugin, whose bundle reads QUALY_ROOT while DSH boots:

export QUALY_ROOT=/absolute/path/to/qualy-ai

node lib/cli.js run start \
  --dsh-root /absolute/path/to/deepseek-harness \
  --dsh-profile qualy \
  --workspace "$QUALY_ROOT" \
  --runs-dir .dsh-test-runs \
  --retain \
  --json

Copy the returned runId, then run the shipped Qualy UI smoke. Later commands reconstruct the DSH adapter from the run manifest, so the DSH flags do not need to be repeated while the runtime is alive:

node lib/cli.js workflow run \
  --file scenarios/qualy-profile-smoke.workflow.json \
  --state-file .dsh-test-runs/qualy-smoke.state.json \
  --runs-dir .dsh-test-runs \
  --run <run-id> \
  --json

node lib/cli.js run stop \
  --runs-dir .dsh-test-runs \
  --run <run-id> \
  --retain \
  --json

The Qualy smoke completes the clean-room onboarding, defers real model credentials, opens the Agent Presets selector, selects Qualy 制造质量智能体, asserts zero console errors and failed network requests, and earns renderer-verified evidence. scenarios/dsh-web-smoke.workflow.json is the profile-agnostic alternative.

If a stopped DSH process must be relaunched, repeat any environment variables required by that profile, such as QUALY_ROOT. Environment values are deliberately not persisted in the public run manifest.

CLI

dsh-test run start
dsh-test run status --run <id>
dsh-test run resume --run <id>
dsh-test run stop --run <id>

dsh-test workflow validate --file <workflow.json>
dsh-test workflow run --file <workflow.json> --run <id>
dsh-test workflow resume --file <workflow.json> --run <id>
dsh-test workflow inspect --file <workflow.json>

Important adapter options:

• --dsh-root <path>: DSH source checkout; selects the built-in adapter.
• --dsh-profile <name>: source profile under $DSH_HOME/profiles (default web).
• --dsh-home <path>: source DSH home (default $DSH_HOME or ~/.dsh).
• --dsh-patch <path>: extra overlay; repeatable.
• --browser-channel <name>: explicit Playwright channel. DSH_TEST_BROWSER_CHANNEL is also supported.
• --host-module <module>: advanced external composition; mutually exclusive with --dsh-root.

The built-in adapter supports these workflow commands:

• dsh.status
• browser.action and aliases such as browser.snapshot, browser.click, and browser.screenshot
• assertion.check / assertion.assert
• evidence.capture

Browser clicks and fills can target a snapshot ref, CSS selector, accessible role plus name, or visible text. Assertions currently expose black-box ui, workspace-confined file, and DSH service facts. Product-specific session and rpc assertions remain available through an external host module.

evidence.capture defaults to a strict renderer gate. It fails the workflow unless it has a non-empty ARIA snapshot, screenshot, console record, network record, and at least one passing assertion. Failed evidence is still retained for diagnosis.

Isolation and recovery

Each run owns:

• manifest.json and idempotent request history;
• an isolated workspace reference and DSH persistence home;
• private browser/runtime state;
• workflow state with a source-definition hash and per-step attempts;
• one immutable directory per evidence capture.

Workflow failures do not advance the cursor. workflow resume retries the failed step with a new attempt identity, while a crash before state persistence reuses the same request identity and receives the recorded response. Editing a workflow after state was written is rejected instead of silently resuming against a different definition.

External host modules

Use --host-module when a plugin needs an internal scaffold, fake model, fake service, or authoritative session/RPC hook that cannot be observed through the public Web surface. The ESM module exports createComposition(context), default.createComposition(context), or testHost.createComposition(context) and returns a TestComposition.

Host modules are an extension seam, not part of a production profile. A host may claim renderer-verified only by returning an evidence manifest inside the run's evidence directory; the controller validates required artifacts, sizes, hashes, run identity, and passing assertions.

Development

pnpm test
pnpm build
pnpm pack:check

The unit suite uses a fake detached DSH launcher. The shipped Qualy workflow was additionally verified against the real local DSH Web UI and system Chrome.