uss7118/dsh-win-control

dsh-win-control - Computer Control

DeepSeek Harness plugin that drives a Windows desktop: program-first automation (window control, UI Automation, browser CDP, local Windows OCR), humanised mouse/keyboard input, and a live control panel.

Self-configuring and adaptive

On first activation the plugin probes the machine and writes its own config to /computer-control/config.json: Windows version, PowerShell version, screen size, OCR engine + languages, UI Automation availability, python/pypinyin. Every later run reuses that file (re-read every few seconds, edits are honoured), and each tool adapts to it:

• no OCR language pack -> computer_find_text says so immediately instead of failing deep
• no python+pypinyin -> typing mode "ime" goes straight to direct Unicode typing
• no UI Automation -> computer_uia explains it; computer_act still uses OCR
• no browser on a CDP port -> computer_dom explains how to launch one
• a Harness UI without the expected slots -> the panel is skipped, Host tools keep working
• computer_selftest re-probes and rewrites the config on demand

Browser pages are read, not scanned

• computer_dom speaks the Chrome DevTools Protocol directly (dependency-free WebSocket

client): read innerText/outerHTML, run JavaScript, click a selector, navigate, or launch Edge/Chrome with an isolated profile and --remote-debugging-port.

• computer_page_text reads the whole page through UI Automation TextPattern, with a local

OCR dump as fallback.

• computer_find_text with an empty text dumps every recognized line of any window.

Prefer HTTP (web_fetch) for public pages and the readers above for logged-in or JS-rendered pages; OCR stays the fallback for canvas, games and self-drawn apps.

Install

Requirements: Windows 10/11 and a DeepSeek Harness build that can install profile bundles. Everything else ships with Windows (powershell.exe, user32/gdi32/imm32, UI Automation, WinRT OCR).

dsh plugin --profile <name> add dsh-win-control

Computer Control (电脑控制)

A DeepSeek Harness plugin that lets the agent — and you — drive this Windows desktop: screen captures, mouse movement/clicks/scroll/drag, and keyboard text or key chords.

One implementation, two callers: every button in the on-screen panel calls the same Host operation that the agent tools call, so the two surfaces cannot drift apart.

Nothing is stored by default

There are three ways to look at the screen, and only the last one touches the disk:

1. Panel preview (you) — a persistent PowerShell worker captures frames into memory.

Host keeps the latest frame in memory, the panel renders it from a Blob object URL and revokes the previous URL as soon as the next frame arrives. Zero disk writes; a frame costs ~90–150 ms, so the panel refreshes at 0.5 s / 1 s / 2 s.

1. `computer_look` (the agent) — the frame is captured in memory, saved as a

*transient* attachment only long enough for one model call, and that object file is deleted immediately after the answer. The agent receives text, never a picture, so nothing accumulates in the attachment library.

1. `computer_screenshot` with `includeImage: true` or `path` — the only explicit

"keep it" option: the picture is attached to the conversation (Harness attachment library, never auto-deleted) or written to the path you name.

Humanised input (default on)

Every simulated input is humanised locally — no model call, no tokens:

• Pointer: relative SendInput moves (MOUSEEVENTF_MOVE) along a quadratic bezier with a

random curvature and 5–14 ms per step, instead of teleporting with SetCursorPos; a 60–180 ms settle follows before the button goes down.

• Typing: three real paths —
• typing: "ime" converts Chinese to pinyin (via the local python + pypinyin, bundled

scripts/pinyin.py), types the pinyin as virtual-key events so the input method composes and shows its candidate window, commits with Space, then OCR-verifies the first characters; if the IME was not composing it undoes (Ctrl+Z) and types the text directly, so the result is always what was asked for.

• typing: "keys" sends virtual-key events only (no Chinese conversion).
• typing: "unicode" injects characters directly (safe for Chinese/emoji without an IME).
• "auto" (default) uses keys for ASCII-only text and unicode when the text contains non-ASCII.

Per-character delay 20–85 ms with an occasional 80–240 ms pause.

typing: "ime" needs a Chinese IME in Chinese mode (press Shift if needed) and a python on PATH with pypinyin; without them the call reports imeFallback and still types the text via unicode. Windows exposes no cross-process IME-state API, which is why the mode verifies the result instead of trusting the mode.

• Wheel: split into 1–3-notch chunks with 25–85 ms gaps instead of one jump.

Pass humanize: false when a deterministic fast path is required (tests, bulk operations). Measured here: 14 humanised characters took ~1.18 s; instant typing takes ~20 ms per character.

Because the Harness window takes the foreground back after each tool result, computer_keyboard and computer_mouse accept window / hwnd and focus that window inside the same call as the input. Without it, a "focus, then type" pair of separate calls can lose focus in between.

Hardware-level limits

User mode cannot fake hardware: SendInput is the deepest documented API, and low-level hooks, Raw Input consumers and anti-cheat can see the LLMHF_INJECTED flag. True hardware-grade input needs a USB HID device (a Pico/Arduino/KMBox style adapter) or a kernel driver such as Interception — both are outside this plugin, and kernel drivers carry real risk.

Execution order: program first, recognition second, vision last

1. Program control — computer_open (shell launch) and computer_window

(list / focus / minimize / maximize / restore / close / move) use Windows APIs directly.

1. Program recognition — computer_uia (UI Automation control tree: probe / tree / find /

invoke / focus / setvalue) and computer_find_text / computer_click_text (built-in Windows OCR on a PrintWindow surface, so a covered window can still be read).

1. Automatic routing — computer_act probes one window and picks the channel itself:

rich UIA tree → trigger the named control with InvokePattern / ValuePattern (no coordinates); thin tree or missing control → local OCR and a click. The result reports channel: "uia" | "ocr".

1. Vision fallback — computer_look for non-text targets (icons, canvas, games) or an

unclear plan.

Browser pages: read them, never scan them

A rendered page is data, not pixels. Three direct readers sit above OCR for that case:

• `computer_dom` — a dependency-free CDP client. It connects to Edge/Chrome over

http://127.0.0.1:/json/list and speaks the DevTools WebSocket protocol directly, so action: "text" returns document.body.innerText, action: "eval" runs JavaScript and returns the value, action: "click" clicks a CSS selector, action: "goto" navigates. Measured here: text read 2 ms, eval 6 ms, launch (isolated profile) 577 ms. Without a browser already started with --remote-debugging-port, action: "launch" starts Edge/Chrome with an isolated profile (default /browser-profile; pass profile to keep it on another drive, e.g. E:\dsh-browser).

• `computer_page_text` — UI Automation TextPattern on the browser document, i.e. the

accessibility tree text rather than the DOM. It works on Electron/Chromium windows too (measuring this Harness page: 7,218 characters in one call, zero OCR). If the window exposes no TextPattern, it falls back to an OCR dump.

• `computer_find_text` with an empty `text` — dumps every recognized line of the window or

region top-to-bottom (a chat pane, a canvas, a game HUD). Measured on a WeChat window: 33 lines in 375 ms, zero model tokens.

Prefer HTTP (web_fetch) for public pages, CDP for logged-in/JS pages, computer_page_text for Electron apps, and OCR only when there is no DOM at all.

Security note on CDP: eval runs arbitrary JavaScript in that browser profile. The debugging port is bound to localhost, so keep it on the loopback interface and use a dedicated profile — do not point it at a profile you also use for banking.

Measured here: UI Automation is rich in Chromium/Electron windows (DeepSeek Harness 1040 named controls, Edge 576) and in native control apps (Calculator 49), but almost absent in self-drawn apps (WeChat 1, classic Notepad 0). computer_act therefore invokes Calculator buttons through UIA in ~1.3 s with no pixels involved, and falls back to OCR clicks for WeChat in ~1.6 s.

Everything uses documented Windows interfaces only: SendInput, GDI capture / PrintWindow, window/process APIs, UI Automation, and Windows.Media.Ocr. No memory reads, injection, API hooking, protocol emulation or credential handling — from the app's point of view it is a normal user clicking and typing.

What it adds

Agent tools:

Tool · What it does computer_act · One operation, automatic route: UIA Invoke/SetValue when the control tree is rich, else OCR + click. Reports which channel ran. computer_uia · Inspect and drive the UI Automation tree: probe, tree, find, invoke, click, focus, setvalue. computer_open · Open a program, file, folder or URL through the shell; returns the launched pid. computer_window · List windows (title, pid, hwnd, rect, minimized); focus / minimize / maximize / restore / close / move one. computer_find_text · OCR the screen or one window; returns matching lines with screen-pixel boxes and centers. With scroll: true it keeps scrolling and re-OCRing until the text appears (pure OCR, no vision). An empty text dumps every recognized line. computer_page_text · Read a whole browser page (or any window with a UI Automation TextPattern) directly: UIA text first, local OCR dump as fallback. computer_dom · CDP direct read: list page targets, read innerText/outerHTML, run JavaScript, click a selector, navigate, or launch a debuggable Edge/Chrome (isolated profile). computer_click_text · OCR-locate text (optionally inside a window, focusing it first) and click its center; scroll: true scrolls until found, then clicks. computer_look · Ask the model about the screen; frame is released right after the call. computer_screenshot · Capture metadata only; includeImage/path are the explicit "keep it" options. computer_mouse · move, click, double_click, right_click, middle_click, scroll, drag. computer_keyboard · type Unicode text or press one key / a chord. computer_screen_info · Monitor bounds, cursor position, foreground window title/rect/pid.

Measured on this machine: computer_find_text on a window ≈ 0.2 s; computer_find_text with scroll: true found a marker 180 lines down after 5 scroll steps (~9.5 s, 0 tokens); computer_act ≈ 1.3 s via UIA (probe + Invoke) and ≈ 1.6 s via OCR; computer_window list ≈ 0.5 s; computer_open ≈ 0.6 s. A computer_look costs ~10–25 s, so it is the fallback only.

OCR details that make this reliable: tokens of one visual line are grouped into rows before matching (Windows OCR splits marker-180 into marker, 一, 180), dash-like glyphs and a CJK 一 between ASCII characters normalize to -, CJK spacing is collapsed, minimised windows are restored, and a PrintWindow capture that comes back blank is retried from the screen.

Web UI: a floating 电脑控制 / Computer Control button in the bottom-right corner of the Harness page opens a panel with a live preview, auto refresh, mouse controls and a text/keys box. The panel calls the Host route /dsh-computer-control/*.

How the agent "sees"

No OCR and no text extraction is involved: the frame is passed to the model as an ordinary image content block, and the model itself interprets it natively (the llm service exposes ImageBlock, inputModalities: text | image and a per-model imageRequestTarget). computer_look therefore returns a model-written description of the screen, with approximate coordinates, and the image object is unlinked before the tool result is returned.

Cost note: a thinking model spends reasoning tokens on the look (a one-line answer took ~3k output tokens and ~20 s in testing). That nested call is not part of the Harness turn accounting.

How it works

• index.js — Host half: the tools, computer_look, the prefix route on

ctx.webServer, the frame worker, the transient-attachment lifecycle, cleanup, and the dependency-free CDP client.

• scripts/screenshot-worker.ps1 — long-lived worker: one Add-Type/DPI setup, then one

JSON line per capture on stdin/stdout, PNG returned as base64.

• scripts/computer-control.ps1 — one-shot helper for mouse, keyboard, screen info,

UI Automation, page text (pagetext) and explicit file/region captures (SendInput + user32).

• client.js — Client half: a shell.overlay entry with inline zh/en dictionaries

registered through the Harness locale service.

Coordinates are physical screen pixels measured from the top-left of the primary monitor (the helpers mark themselves per-monitor DPI aware).