dsh-qa

DSH QA

Explore real apps, capture evidence, and turn verified actions into repeatable QA scenarios.
_{Agent-Led Exploration • Evidence-Backed Assertions • Deterministic Replay • Browser, Desktop, iOS & Android}

_{Package: @zseven-w/dsh-qa · Version: 0.1.0-rc.2 · Prerelease}

English · 简体中文

Quick start · Safety and limitations · Known gaps · Development · Documentation

_{Actual output from the shipped browser example: two steps and final assertions passed. This is a typeset view of the unedited Markdown report, not a built-in dashboard or a claim of four-platform coverage.}

Why DSH QA

QA orchestrator plugin for DeepSeek Harness: an agent explores your App like a real user (with evidence-backed findings), then the explored path is exported as a deterministic Replay scenario that runs on every release.

• Explore mode — the agent drives the app through a tool surface (qa_session_start / qa_observe / qa_act / qa_assert / qa_evidence / qa_record_export / qa_replay_run / qa_session_stop) and documents what it sees.
• Replay mode — declarative QaScenario files (lossless JSON) executed deterministically with per-step re-observe assertions, producing redacted JSON / Markdown / JSONL reports.
• Drivers — @zseven-w/dsh-browser (BU, contract v9), @zseven-w/dsh-computer (CU, contract v5), @zseven-w/dsh-ios, and @zseven-w/dsh-android. Driver safety semantics are inherited, never loosened: EXTERNAL_COMMIT_TARGET refused, secure fields permanently refused, approval gates passed through, unknown receipts require re-observation. Mobile sessions require an explicit device id and never fall back to a default device. Mobile text input is conditional, never blanket-implemented: iOS fill/type use the dsh-ios native element-bound fillTarget/typeTarget primitives when the live driver exposes them, with missing methods or missing native identifiers left explicitly unavailable and no raw global-type fallback; Android type remains append-faithful after real focus verification and Android fill remains FILL_PRIMITIVE_UNAVAILABLE.

QA coordinates four independent drivers; it does not replace them or grant broader access. A passing fixture, unit suite, or device acceptance run is evidence for that tested scope, not a claim that every app is supported.

Quick start

Requires Node.js 24.11.0 or later. QA orchestrates drivers it does not contain: it loads each one lazily, by package name, only when a session asks for that platform. Installing @zseven-w/dsh-qa alone gives you no drivers — qa_session_start will report the driver as not installed. Install the ones you need alongside it:

npm install @zseven-w/dsh-qa           # the orchestrator
npm install @zseven-w/dsh-browser      # browser sessions
npm install @zseven-w/dsh-computer     # macOS desktop sessions
npm install @zseven-w/dsh-ios          # iOS device/simulator sessions
npm install @zseven-w/dsh-android      # Android device/emulator sessions

Verify the install by replaying the shipped browser example, which needs nothing but dsh-qa, dsh-browser, and a local browser:

node node_modules/@zseven-w/dsh-qa/scripts/run-example.mjs

It serves the packaged web fixture on an ephemeral loopback port, replays a two-step scenario headlessly, and writes JSON / Markdown / JSONL reports. A working install prints [example] status: pass. See Running the shipped example for the details.

For DSH-host usage, install or update DSH with:

npm install -g @deepseek-ai/dsh@latest

Installing DSH does not activate this plugin. Its host entry is declared in cordis.patch.yml; the standalone stdio MCP entry is src/server.mjs, also exposed by npm run mcp and .mcp.json. Configure the chosen host to load the plugin/server and provide the required drivers before starting a session. Follow the Explore playbook for the observe → act → assert → evidence → export workflow.

Safety and limitations

• No false green: unknown action receipts need fresh proof. Runs report pass, inconclusive, or fail; missing coverage is not absence.
• No authority escalation: driver approvals remain in force. Secure fields and EXTERNAL_COMMIT_TARGET are refused; mobile sessions require an explicit device id, with no default-device fallback.
• Replay needs durable targets: coordinates, ephemeral references, and ambiguous selectors are not promoted into durable scenarios. An action with no settled, provable outcome is excluded from export.
• Vision is advisory: visual assertions assist triage but do not determine the run status. Model narration is not observed fact.
• Mobile support is conditional: iOS text input requires live native element-bound primitives and identifiers. Android type verifies real focus; Android fill remains unavailable.
• Evidence needs care: structured reports use fail-closed redaction, but screenshots can still contain private content. Use synthetic test data and inspect artifacts before sharing. Login-state injection requires explicit owner authorization for exact origins; see login state.

Known gaps

This is a developer release. The main path — Explore → evidence → Export → Replay on browser and desktop — is exercised by the suite on every change, but these are open, and knowing them is part of using the package honestly.

• The Computer helper is built by you, by design. It ships ad-hoc signed with no TeamIdentifier and no stapled ticket: you build it from the dsh-computer checkout and grant it Accessibility + Screen Recording yourself. This is a settled decision, not a pending task — there is no Developer ID notarization planned, so do not wait for an "install and go" desktop build. The browser, iOS and Android drivers are unaffected.
• Absence is often unproven rather than proven, and that shows up as inconclusive. node-absent passes only when the deciding view is complete AND the browser driver's bounded closed-shadow-root probe verified its coverage. When that probe finds a closed root, exceeds its node budget, or cannot open a CDP session, the assertion carries COVERAGE_UNVERIFIED or INCONCLUSIVE_TRUNCATED and the run is inconclusive — never pass. On shadow-DOM-heavy or very large pages, expect that instead of a green absence. fail is reserved for a real defect: a node that IS observed disproves node-absent and fails normally, whatever the coverage state.
• Deep targets on large pages stay inconclusive. Beyond the driver's 100-node observation window, a scoped scroll proof can only reach INCONCLUSIVE_SCOPE, never pass. This is honest, not broken — but it means deep flows on big pages do not produce a green gate today.
• Visual assertions are advisory and the live vision path is unverified here. Vision never changes a Replay's pass/fail by design. The seam to a real host vision service (ctx.llm / attachments) is covered only by a fake in the suite; it has not been run against a live vision model.
• Mobile drivers carry no contract version. The browser and computer drivers publish a contract version and QA now REFUSES to load one that is not the version this build supports (browser 9, computer 5) — older and newer alike, because a contract change can alter what an observation means rather than merely adding to it. dsh-ios / dsh-android export no version from /driver, so QA still loads them structurally and a drift there is caught after the fact rather than at load.

Replay verifies five kinds of semantic assertion. A pass means those assertions held on fresh observations — not that the app is correct, not that the screen looks right, and not that coverage was complete.

Implementation reference