dsh-click

🖱️ dsh-click

• 1024 store channel: npm i -g dsh1024 once, then dsh1024 plugin --profile web add dsh-click (counts toward the deepseek1024.com install ranking).

Cross-platform native desktop control for DeepSeek Harness — Windows first.

*Look at the screen, then act — every click gated, every action audited.*

English · 简体中文 · Español · Português · हिन्दी

dsh plugin --profile <name> add dsh-click

📖 Ecosystem knowledge base — measured data, not marketing: plugin development guide · plugin-selection data · maintenance criteria.

Install & uninstall

• git channel (latest main): dsh plugin --profile web add github:PerryLink/dsh-click — the prepare script builds with production dependencies only.
• npm channel (published releases): dsh plugin --profile web add dsh-click.
• tarball channel: pnpm pack in this repo, then dsh plugin --profile web add ./dsh-click-.tgz.
• uninstall: dsh plugin --profile web remove dsh-click (or remove the row from the profile patch).

If pnpm reports ERR_PNPM_IGNORED_BUILDS for this package (esbuild's harmless platform-binary validation), add allowBuilds: { esbuild: true } to your pnpm-workspace.yaml — the dsh CLI prints the exact snippet.

⭐ 如果它帮到了你

这个插件是 DSH 插件家族的一员（95 个，全部 Apache-2.0）。如果你在用，给个 star —— 它不会解锁任何功能，但会让下一个人在搜索里更容易找到它。

*English:* part of a 95 plugin family for DeepSeek Harness. If it is useful, a star helps the next person find it — nothing is gated behind it.

What is dsh-click?

Cross-platform native desktop control for DeepSeek Harness — Windows first.

Look at the screen, then act — every click gated, every action audited.

*Workflow overview: install → open panel → act. The examples below use the real command surface.*

Compatibility

Surface · Status Harness · DeepSeek Harness dsh-v0.2.1-alpha.1 (GitHub tag, verified 2026-09-25). Verified 2026-09-11 against the dsh-v0.1.7-alpha.1 master checkout (full gate chain + profile install smoke). Node · ^22.19.0 \ · \ · >=24.0.0 Platforms · Windows first (UIAutomation + Win32 input, via a bundled PowerShell helper); macOS/Linux backends are reserved and fail closed with a clear reason Model · Text-only models fully supported (screen_read returns structured text); vision models additionally get screen_shot images

What you get

dsh-click gives the harness a complete observe → act loop over native desktop applications:

• `screen_shot` — screenshot of a window (or the primary screen), downscaled to a configurable bound. With a vision-capable model the result carries the image; otherwise a text description keeps text-only models working.
• `screen_read` — the structured observation: the window's accessibility tree (element ids, types, names, rectangles, supported patterns) plus pixel-location hints with colors — plain text, no image model required.
• `click` / `type` / `scroll` / `key` — window-scoped actions addressed by element id or coordinates. Delivery prefers UIA invoke, falls back to posted window messages — and never steals foreground focus.
• `app_list` / `app_launch` — enumerate running applications and their windows; launch one by name or path.

Every mutating action crosses one safety boundary:

1. Freshness — the action must cite a basedOn observation; the window is re-captured right before acting and the action is refused if the screen changed (pixel-hash check + max-age bound).
2. Approval — ctx.approval gates every action by default; window-title/executable regexes can allowlist specific windows (still audited).
3. Process identity — the owning process's pid and executable path are verified before and after the act; a change refuses the outcome loudly.
4. Audit — observations and actions land in the session log as dsh-click/observed / dsh-click/action events (sanitized, log-only).

model                           harness
  │ screen_read ──▶ observationId (+ elements, pixels)         ← structured text
  │ click {basedOn, target} ──▶ freshness check ──▶ approval ──▶ helper (UIA)
  │                             pixel hash changed? ── refuse + re-observe
  │                             pid/exe changed after act? ── PROCESS_CHANGED
  │ ◀── canonical JSON + audit events (dsh-click/action)

Quick start

dsh plugin --profile web add github:PerryLink/dsh-click

# 1. install the bundle into your profile
dsh plugin --profile web add github:PerryLink/dsh-click

# or from npm (published releases)
dsh plugin --profile web add dsh-click

# 2. restart and verify the row
dsh --profile web --dump-config | grep -A2 'id: dsh-click'

Then ask the agent to look at a window and act — the approval prompt appears for every mutating action:

> Open Notepad, type "hello", then read back what is on screen.

Examples

These are the plugin's real entry points; install once, then call one.

Detailed example

Command

~~~sh app_list ~~~

Input

~~~json {} ~~~

Output

~~~text 1 running application(s) with windows:

• Demo (pid 4242, C:\Apps\demo.exe)
• [windowId 1122867] "Demo App" at (10, 20) 800x600

~~~

*Observed from plugins/dsh-click/tests/tools.spec.ts:159-185; renderer at src/tools.ts:933-942*

Failure branch

Input

~~~text {} ~~~

Output

~~~text Error: dsh-click backend unavailable on linux: Linux backend is reserved but not implemented yet (Windows-first); only Windows can act today ~~~

*Observed from plugins/dsh-click/src/platform/selection.ts:61-62,81; plugins/dsh-click/src/tools.ts:133-136; plugins/dsh-click/tests/tools.spec.ts:226-245*

Brief examples

• screen_read
• app_list

Configuration

All tunables are Schemastery Config fields (changeable from cordis.yml). An id-targeted override replaces the whole row — restate every key you need. cordis.patch.yml documents each key inline.

Key · Default · Meaning requireApproval · true · Gate every mutating action behind approval; observers never ask autoApproveWindows · [] · Window-title/executable regexes that skip the approval ask (still freshness-checked and audited) auditSessionEvents · true · Append dsh-click/observed / dsh-click/action session audit events. The adaptive gate already skips the append on envelope-less hosts (rc.6–rc.8, 0.1.1-rc.2, and 0.1.2-rc.1, which fails closed on unknown types at read); set false to stop audit appends entirely 0.1.2-rc.1 (adapted 2026-09-02): the session envelope keeps its ignorable field for stored-log read compatibility only - Session.append still cannot stamp it, so audit-gate behavior is unchanged. focusFallback · never · Whether an action may bring the target window to the foreground as a last resort (never / allow) imageMode · auto · screen_shot rendering: auto (image when the model accepts images, text otherwise) or text helperTimeoutMs · 30000 · Per-helper-call timeout in ms (1..300000) maxHelperOutputBytes · 25165824 · Cap on one helper response in bytes (1024..67108864) maxScreenshotSide · 2560 · Longest screenshot side in pixels (320..7680); larger captures are downscaled staleCheckPixels · true · Compare a fresh pixel hash before every action and refuse on change maxObservationAgeMs · 30000 · Maximum age in ms of an observation an action may cite (1000..600000) maxCachedObservations · 8 · LRU cap on cached observations (1..64) maxElements · 500 · Cap on accessibility elements per screen_read (1..2000) maxTreeDepth · 32 · Maximum accessibility tree-walk depth (1..64) maxTextLength · 200 · Truncation length for sanitized model-visible strings (16..10000) rollbackEnabled · true · Back up and restore control text when type fails ocr.enabled / command / language · true / tesseract / eng · Optional OCR for the screen_find path (probed at mount; degrades to unavailable when tesseract is absent)

Example override in your profile patch:

- insert:
    - id: dsh-click
      name: dsh-click
      config:
        requireApproval: true
        autoApproveWindows: ['^Notepad']
        focusFallback: never

Tools & surfaces

Tool · Read-only · Needs approval · Notes screen_shot · ✅ · — · Returns an observationId later actions cite in basedOn; image attachment when the model accepts images screen_read · ✅ · — · Accessibility tree + pixel hints; element ids are what actions address click · · ✅ · Exactly one of elementId or (x, y); UIA invoke preferred, posted messages fallback type · · ✅ · Value-pattern elements only; backs up and restores control text on failure scroll · · ✅ · Element (scroll pattern) or window (posted wheel) key · · ✅ · Posted key combinations ("Ctrl+S"); apps that ignore posted input refuse loudly app_list · ✅ · — · Running applications and their visible windows app_launch · · ✅ · By name or executable path, with optional arguments

Permissions & data

• Permissions: mutating actions cross the official ctx.approval seam — the plugin never re-implements or bypasses it. The allowlist only ever *skips the ask for specific windows*; it cannot disable the freshness or process-identity checks.
• Data: the plugin stores nothing on disk except the screenshots the attachment store keeps (content-addressed, under the harness's own attachment policy). Observations are cached in memory (LRU, bounded). No network requests, no credential storage.
• Session log: dsh-click/observed and dsh-click/action are log-only audit events carrying sanitized window/process facts — titles, paths, and free text are redacted and length-capped before they are written or shown.

Security boundaries

• Observe before act, every time. Actions must cite a fresh observation; a changed screen (pixel hash) or an expired observation is refused with a model-readable reason demanding re-observation.
• Approval is the default. requireApproval: true unless you explicitly opt specific windows in; every action — allowed or not — is audit-logged.
• No foreground stealing. The helper never brings a target window to the foreground (focusFallback: 'never' by default); input is delivered through UIA or posted messages so background windows are not disturbed.
• Process identity is re-verified immediately before and after each action; a mid-act process swap fails the outcome (PROCESS_CHANGED).
• Sanitized output. Control characters are stripped, tabs collapse, credential-shaped values (keys, tokens, JWTs, bearer headers) are redacted before anything reaches the model or the log.
• Fail closed. Unsupported platforms, a missing subprocess service, or an unavailable helper refuse every call loudly — profiles keep booting everywhere.

Known limitations

• Windows first. macOS and Linux backends are reserved; on those platforms every call fails closed with a clear reason.
• Text-only fidelity. screen_read depends on the application exposing UIAutomation; apps without an accessible tree yield pixel hints only. Coordinate clicks remain available.
• Posted-input apps. Some applications ignore posted window messages (games, some Electron surfaces); key reports this honestly instead of pretending success.
• Session audit on envelope-less harness builds. The audit events ride an adaptive gate: hosts that know the vocabulary append plainly, hosts with the ignorable envelope append with the marker, and envelope-less hosts — 0.1.0-rc.6–0.1.0-rc.8, 0.1.1-rc.2, and 0.1.2-rc.1 (which cannot stamp the ignorable marker — the envelope field is retained for stored-log read compatibility only — and fails closed on unknown types at read) — get no audit append; the tool results remain the reconstructable audit trail. Set auditSessionEvents: false to stop audit appends entirely.

Development

pnpm install        # node ^22.19 || >=24
pnpm run typecheck  # tsc: src + tests against the local harness checkout
pnpm run typecheck:ci  # tsc against the published 0.1.7-rc.2 types (no paths)
pnpm test           # vitest: 79 tests, 13 files (helper smoke runs on Windows)
pnpm run build      # tsdown bundle + tsc declarations (lib/)
pnpm run verify:self-contained  # dependency specs resolve from the registry
pnpm run verify:artifacts       # built ESM face + native helper present
pnpm pack           # the published tarball

Topics

dsh, dsh-plugin, deepseek-harness, deepseek, cordis, computer-use, windows-automation, uiautomation, desktop-control, screen-reader

Contributors

• Windows helper smoke checklist — real input/focus verification for 0.3.24+.
• @LIU12312-Source — real SendInput/foreground-focus fallback for the Windows helper (PR #21).
• @PerryLink — creator and maintainer: tool surface, action safety boundary, Windows native helper, sanitizers, and the five-language docs.
• @Mchsd — added the auditSessionEvents opt-out for harnesses whose session reader rejects the dsh-click audit events (#2).

PerryLink DSH Plugin Family

This project is part of the DeepSeek Harness plugin family from PerryLink. The roster stays collapsed below so every README keeps a short first screen; expand it to see every row and its status.

Expand all plugins

Plugin · Status dsh-aqua-input-check · dsh-archive-check · dsh-auto-review · dsh-autotier · dsh-bid-ca-precheck · dsh-bid-qual-check · dsh-cert-mcp · dsh-checkpoint-rewind · dsh-click · dsh-composer-history · dsh-contract-stance · dsh-customs-doc-check · dsh-data-quality · dsh-demurrage-ledger · dsh-doublecheck · dsh-drill-script-check · dsh-eia-guide-check · dsh-emergency-plan · dsh-essay-rubric-check · dsh-evidence-check · dsh-export-ctl-check · dsh-fast · dsh-fmea-table-check · dsh-forecast-penalty · dsh-fund-research · dsh-github · dsh-gongwen-flow-check · dsh-gongwen-word-check · dsh-guard-plan-qc · dsh-hazchem-check · dsh-hazplan-check · dsh-hidden-risk-map · dsh-hs-classify · dsh-icd-rule-check · dsh-industry-research · dsh-lawcite-adapter · dsh-laya · dsh-lc-doc-check · dsh-learning-gap-check · dsh-library · dsh-local-ai · dsh-lsp-actions · dsh-mask · dsh-mcp-panel · dsh-medrec-qc · dsh-memento · dsh-nsfc-form-check · dsh-nurse-record-check · dsh-observe · dsh-origin-rvc-check · dsh-ota-review-check · dsh-output-styles · dsh-paper-doc-adapter · dsh-permission-rules · dsh-permit-report-check · dsh-pipeline-check · dsh-pleading-draft · dsh-plugin-doctor · dsh-plugin-guide · dsh-plugin-kit · dsh-plugin-upgrade · dsh-policy-brief-draft · dsh-power-loss-split · dsh-power-ticket-check · dsh-ppap-check · dsh-protest-deadline · dsh-railway-window · dsh-repair-order-qc · dsh-research-report · dsh-review-reply-check · dsh-rulefile-check · dsh-safety-brief-check · dsh-score · dsh-session-sync · dsh-site-log-check · dsh-skill-pack-security · dsh-soe-decision-check · dsh-soilwater-check · dsh-sop-sync-check · dsh-spc-gbt-adapter · dsh-talk · dsh-tender-extract · dsh-tender-matrix · dsh-test-drive · dsh-ticktick · dsh-translate · dsh-warranty-calc · dsh-budget · 🧊 FROZEN — see the repo README dsh-claude-move · 🧊 FROZEN — see the repo README dsh-defend · 🧊 FROZEN — see the repo README dsh-draw · 🧊 FROZEN — see the repo README dsh-reach · 🧊 FROZEN — see the repo README dsh-background-agents · 🚫 RETIRED — see the repo README dsh-session-pin · 🚫 RETIRED — see the repo README dsh-team-rooms · 🚫 RETIRED — see the repo README

License

Apache License 2.0 © 2026 dsh-click contributors