dsh-xiangwriter-screen-control

dsh-xiangwriter-screen-control

Windows screen control for DeepSeek Harness. Six screen_* tools let an agent look at this desktop and drive it like a person: screenshot, move/click/drag/scroll the real mouse, type Unicode text, send key chords, read the UI Automation tree, focus windows, and wait for UI changes.

Like Codex's computer-use plugin, but built for DSH on Windows with no dependencies, no MCP server, no compiler: a single Windows PowerShell 5.1 worker does the Win32 work, and the agent gets first-class tools, approval gating, and an emergency stop.

中文说明 · Tools · Architecture · Verification

---

Features

• Look — full virtual desktop, one window (works while covered, via PrintWindow), or a rectangle; PNG/JPEG with optional downscale and a cursor crosshair.
• Act — click, double/right/middle click, drag with interpolation, wheel scroll, Unicode typing (Chinese, emoji), key chords such as ctrl+shift+t.
• Understand — UI Automation tree with control type, name, automation id, rectangle, enabled/off-screen state and values; window list with process names and rectangles.
• Image-aware — when the routed model accepts images, the screenshot is committed through the attachment service and returned as a real image block; text-only routes get the file path plus metadata instead.
• Safe by default — desktop input is gated by inputPolicy: "ask" (one approval per session), Ctrl+Alt+Esc stops all input for 30 s, and every call reports the cursor position it actually read back.
• Cheap to run — one long-lived worker per profile, serialized requests (~100 ms per action after the first), automatic restart, one-shot fallback when the host forbids piped stdio.

Requirements

• Windows 10 or Windows 11 with an interactive desktop session.
• DeepSeek Harness (tested with 0.2.1-rc.2).
• Windows PowerShell 5.1 (powershell.exe), which ships with Windows. PowerShell 7 is not required.
• Optional: a model that declares image input, for screenshot understanding. Text-only models can still work through screen_ui.

Install

From GitHub (recommended)

dsh plugin add github:xiangwriter-dev/dsh-xiangwriter-screen-control

From a local checkout

git clone https://github.com/xiangwriter-dev/dsh-xiangwriter-screen-control.git
dsh plugin add /absolute/path/to/dsh-xiangwriter-screen-control

In the desktop app, the equivalent is Settings → Plugins → Install bundle with the directory as the target.

From a release tarball

dsh plugin add https://github.com/xiangwriter-dev/dsh-xiangwriter-screen-control/releases/latest/download/dsh-xiangwriter-screen-control.tgz

After installing, the tools appear on the next turn. The plugin row is called Screen Control in Settings → Plugins.

From the plugin market (Discover)

DSH 0.2.1-rc.2 ships the plugin market as an experimental bundle that is disabled by default, and it has no built-in index: the Discover tab reads the index URLs you add. Enable the bundle once (Settings → Plugins → Plugin market; the package is @deepseek-ai/dsh-experimental-plugin-market-bundle), then add this index:

https://raw.githubusercontent.com/xiangwriter-dev/dsh-xiangwriter-screen-control/main/market/index.json

The index uses the harness's plugin market format 1 and pins the v0.1.0 commit, so installing from Discover gets exactly the released code. See docs/market.md for the entry, the update procedure, and the exact validation rules.

Quick start

Ask the agent for an outcome, not a keystroke:

Open Notepad, type a haiku about the sea, and screenshot the result.

A typical tool sequence:

1. screen_windows { action: "list", title: "记事本" } — is the target already open?
2. screen_capture { window: "记事本" } — look before acting.
3. screen_pointer { action: "click", x: 640, y: 520 } — click the text area.
4. screen_keyboard { action: "type", text: "…" } — type.
5. screen_capture { window: "记事本" } — verify the text is really there.

Pixels picked from a screenshot can be reused directly:

// 1) capture returns: source {x:0,y:0,width:2560,height:1600}, scale 0.625
{ "name": "screen_capture", "arguments": { "max_width": 1600 } }

// 2) click the pixel you saw in that image — the plugin maps it back to the screen
{ "name": "screen_pointer", "arguments": { "action": "click", "x": 400, "y": 300, "coordinate_space": "image" } }

Tools

Full parameter and return contracts: docs/tools.md.

Configuration

The plugin row accepts these keys (all optional; defaults shown):

inputPolicy: ask            # ask | allow | deny
capture:
  maxWidth: 1600            # downscale screenshots to this width
  format: png               # png | jpeg
  quality: 82               # jpeg quality
  keepShots: 20             # reserved for capture retention
failsafe:
  enabled: true
  hotkey: ctrl+alt+esc
  cooldownMs: 30000
bridge:
  command: ''               # override the powershell.exe path
  timeoutMs: 15000          # per-action timeout
  restartLimit: 2           # consecutive worker restarts allowed
  idleTimeoutSec: 1800      # worker exits after this much idle time

inputPolicy:

• ask — default. The first pointer or keyboard action in a session asks for approval; the grant is remembered for that session.
• allow — no prompts (only for fully trusted sessions).
• deny — read-only: screenshots, window and UI Automation queries still work, input is refused.

Safety and privacy

• Approval gate. Desktop input is what a person's hands do; the plugin routes the first input action of a session through the harness approval channel. Screenshots and UI trees never prompt.
• Emergency stop. Hold Ctrl+Alt+Esc while any input tool runs (or before it) and input is refused for 30 seconds with E_FAILSAFE.
• Elevated windows. Windows' UIPI blocks synthetic input to processes at a higher integrity level (an administrator app). Run the harness at the same integrity level if you need to drive one; the plugin reports E_UIPI when it can detect the refusal.
• Locked screen. The secure desktop cannot be captured or driven; expect E_LOCKED.
• Screenshots contain everything on screen, including other windows. The plugin has no redaction; do not use it on a desktop with content you would not share with the model provider.
• The worker never touches the network. It receives JSON lines on stdin and returns capture files under your temp directory.

Troubleshooting

Development

node scripts/check.mjs                  # imports every module and validates resources (no child process)
node scripts/run-tests.mjs              # unit tests
node scripts/run-tests.mjs --sandboxed  # unit tests in one process (confined host)
DSH_SCREEN_E2E=1 node scripts/run-tests.mjs e2e --sandboxed

run-tests.mjs enumerates the test files itself, so it does not depend on shell globs or directory arguments; --sandboxed avoids spawning one process per file, which confined hosts forbid.

The e2e suite drives the real engine and briefly moves the mouse; it restores the cursor when it finishes. On CI it is skipped because the runner has no interactive desktop.

node scripts/setup-repo.mjs <github-owner>   # re-point metadata if the repo moves
node scripts/make-entry.mjs <github-owner>   # regenerate the awesome-dsh-plugin entry

How it works

flowchart LR
  A[Agent tool call] --> B[screen_* tool]
  B --> C{Approval gate<br/>input only}
  C -->|allow| D[Node bridge<br/>JSON lines]
  C -->|reject| X[error to model]
  D --> E[Windows PowerShell 5.1 worker]
  E --> F[Win32: user32 / gdi32 / UIAutomation]
  F --> G[Desktop: windows, mouse, keyboard]
  E --> H[PNG/JPEG file]
  H --> I[attachments.saveImage]
  I --> J[image block + metadata]

More detail, including the JSON protocol and lifecycle rules, is in docs/architecture.md.

License

MIT.