dsh-real-time-computer-use

dsh-computer-use

Computer Use for DeepSeek Harness on Windows: an agent can see the Windows desktop and operate it like a person — full-screen capture, look back at any frame of the last ~20 minutes by timestamp, and mouse/keyboard injection.

Architecture

Two halves talking over stdio, one JSON request per line:

The plugin starts the helper once and keeps it resident (a fresh process per frame would pay tens of milliseconds of GDI/DXGI initialization every time). Frame files land in the Windows temp directory and are read back through /mnt/<drive>/....

Requirements

• DSH running in WSL; the Windows helper is launched through WSL interop. A pure-Windows DSH install does not work as written, because Windows paths are mapped as /mnt/<drive>/....
• Windows with .NET Framework 4 (present on every supported Windows). No SDK, no ffmpeg, no third-party binary.
• Screen resolution decides the tile geometry: tiles are sized to fit the harness image budget, so a 2560×1440 desktop yields 2×3 tiles of 1280×480 (614,400 px each, delivered unscaled).

Install

# 1. clone the repository somewhere permanent, then link it into the profile
git clone https://github.com/1azybug/dsh-real-time-computer-use.git
pnpm dsh plugin --profile web add link:"$PWD/dsh-real-time-computer-use"
# 2. restart dsh web — host-layer plugins are only read at startup

Install the operating discipline as a skill (the package ships an English SKILL.md; nothing installs it for you):

cp -r skills/computer-use ~/.dsh/skills/      # or $DSH_HOME/skills/

Tools

Capture and inspection: screen_grid (one capture → thumbnail + 1:1 tiles, optionally atSeconds for a frame from the buffer), screen_frames (pull a time range out of the recording buffer; every image carries its capture time), screen_watch (start/stop/stats of the continuous capture), screen_windows (window under a point, foreground window, or full enumeration), cursor_state.

Actions: mouse_move_to, mouse_move_by, click, mouse_button, drag, scroll, type_text, press_key, key_state, hotkey, wait.

Optional tools stay unregistered unless enabled in configuration: wait_for_change / act_when (triggerTools), screen_observe / region_observe (singleImageTools), screen_diff (diffTool).

Configuration

Every knob lives in cordis.patch.yml, and the shipped defaults are not a neutral baseline: they are the configuration one GUI-evaluation setup runs with. Out of the box you get a 16-tool surface; switching the four optional groups on widens it to 21 tools and re-enables ordinary image reading.

Read those four defaults as answers to four questions, not as arbitrary flags:

Why those defaults: on a 2560×1440 desktop a single full-screen image is either downscaled past legibility (a 46 px control becomes 19 px) or covers one corner, so the plugin standardizes on one full-view form — screen_grid, one capture turned into a same-frame thumbnail plus 1:1 tiles. Letting read_image back in would restore the single-image path for any image, not only screen captures. The triggerTools entry is an evaluation constraint (watching for a visual condition and then acting counts as cheating in that setting), not a technical limit. The comments in the file record each decision.

The remaining keys are ordinary capture settings:

The plugin's own code defaults differ for frameIntervalMs (33), backend (gdi) and codec (jpeg): the patch file is where a deployment states its choice, and the code keeps the conservative value for deployments that mount the plugin without it. denyReadImage is true in both.

Host APIs used

The plugin registers against the harness ToolService (ctx.tools.register, ctx.tools.guard, ctx.tools.restrict) and Cordis ctx.effect for ownership.

Verified by a live run on a clean install: harness @deepseek-ai/dsh@0.1.7-rc.1 (the newest published release at the time of writing) in a fresh DSH_HOME, this repository cloned from GitHub and linked into the profile — the plugin loads and all 16 tools register. The same symbols are also present in @deepseek-ai/dsh-tools@0.1.5-rc.3.

Limits

• The desktop is a single shared resource. Keyboard and mouse are global: two DSH instances on the same machine must not act at the same time, or their actions interleave.
• Capture runs at the rate fixed by frameIntervalMs (40 fps in the shipped patch) and costs roughly a quarter of one core while active. The rate is a deployment choice, not a tool argument: screen_watch has no fps parameter.
• The per-frame interval bound is met at the shipped rate, with a rare exception. The target is that no frame interval exceeds 33.333 ms; the shipped frameIntervalMs: 16 (62.5 fps) leaves 17.3 ms of headroom, so the once-per-segment encoder pre-build (~220 ms of Media Foundation work started concurrently with capture, costing the capture thread 10–15 ms) no longer crosses the bound. Measured under a full-screen animation load with a second helper recording alongside: 1 frame of 11217 at 16 ms, versus 4 of 2408 at 25 ms; that single miss was a system-level spike (64.7 ms), not the pre-build. Sealing (Finish+Dispose, 176 ms under load) runs on its own thread. Zero misses are not guaranteed — a system-level stall can still exceed the bound at any rate. A tested alternative (resident pre-build thread, capture at AboveNormal / background at BelowNormal) only moved the same stall to a different frame and made the worst case worse, so it was not adopted. Numbers/conditions: helper/README.md.
• H.264 segments are lossy (PSNR 44.8–50 dB); it does not affect reading small text, but it is not lossless.
• The helper must be recompiled with the Windows-shipped csc.exe if you change helper/*.cs; the build command is in helper/README.md.