asdasdsdsdasdasdasd/dsh-computer-use

dsh-computer-use

A DeepSeek Harness plugin that lets the agent see and operate a Linux X11 desktop: screenshot, move/click/drag/scroll the mouse, type text, and send key combos.

The Linux/X11 computer-use plugin for DeepSeek Harness. Most computer-use plugins target macOS or Windows — if your agent runs on a Linux box with a real graphical session, this is the one that works, with zero heavy dependencies (no Node native modules, no PowerShell, no accessibility framework).

This is not an official DeepSeek package. It drives the real pointer and keyboard through XTest. Treat it as full desktop control.

Which computer-use plugin is right for you?

Pick dsh-computer-use if your agent runs on Linux X11. No Node build step, no PowerShell, no accessibility framework — just python3, libX11, libXtst, and ImageMagick.

What it adds

See examples.md for end-to-end usage.

Coordinates are native X11 pixels, not pixels of a downscaled screenshot. Every screenshot reports scale (and region offset) so:

screen_xy = region_xy + image_xy * scale_xy

Requirements

Linux X11 only. Not Wayland, macOS, or Windows.

• DeepSeek Harness (tested against 0.1.0-rc.8)
• A graphical session with DISPLAY set
• python3
• libX11 and libXtst (libx11-6 / libxtst6 on Debian/Ubuntu)
• ImageMagick (import, convert, identify)

Debian/Ubuntu:

sudo bash scripts/install-deps.sh   # apt install python3 libx11-6 libxtst6 imagemagick
python3 xc.py info

xc.py info should print JSON with screen width/height and "xtest": true.

How it works

The plugin ships a tiny Python helper xc.py that talks to the X server through ctypes (libX11 + libXtst), so it needs no external Python packages and no Node native modules. Every input action reports the native pixel coordinates the OS actually registered, and every input action can optionally attach a fresh screenshot (observeAfterAction) so the model can immediately verify the result. See architecture for the full design.

DSH agent ──► index.js (Cordis tools) ──► xc.py (ctypes/libX11+libXtst) ──► X server ──► your desktop

Install

Into an existing profile (replace web if you use another):

dsh plugin --profile web add github:asdasdsdsdasdasdasd/dsh-computer-use

Pin a commit so later pushes cannot change what you run:

dsh plugin --profile web add github:asdasdsdsdasdasdasd/dsh-computer-use#<sha>

From a local checkout:

dsh plugin --profile web add .

Restart the DSH process after install. Confirm the layer with:

dsh --profile web --dump-config

You should see a # == dsh-computer-use layer and a computer-use row.

If you already use dsh-tool-result-dedup

Identical computer_* results (same click, same key) get collapsed as “omitted to save tokens”, which confuses the model. Skip that prefix in your profile cordis.patch.yml:

- id: tool-result-dedup
  config:
    skipNamePrefixes:
      - computer_

Security

Default sandboxMode is danger-full-access. The agent can click, type, and send shortcuts on your real desktop (including password fields, other apps, and destructive actions).

Only enable this plugin on a machine and account you are willing to let an LLM operate. Do not install it on a shared kiosk or a session with secrets on screen.

Configuration

Override in the profile cordis.patch.yml (later layers replace the whole config object):

- id: computer-use
  config:
    helper: /absolute/path/to/xc.py   # default: xc.py next to index.js
    shotDir: /tmp/dsh-computer-use
    defaultWidth: 1280
    sandboxMode: danger-full-access
    workspaceRoot: .
    observeAfterAction: true          # attach a screenshot after input actions
    nudgeAnnouncedTools: false

:warning: This is full desktop control. Read SECURITY.md before enabling it on any machine you care about.

License

MIT