dsh-pve-vm

dsh-pve-vm

Proxmox VE VM control for DeepSeek Harness: see a VM's screen and click in its console over VNC, plus commands, files and typed text through the QEMU guest agent.

Almost every computer-use plugin drives the desktop the agent itself runs on — and therefore needs X11 or Wayland, a screenshot tool and an input injector on that host. This plugin does something different: it drives the console of a virtual machine on a Proxmox VE node, over one ssh connection. The host can be a headless server, a Wayland desktop or a laptop in another country; the guest can be Windows, Linux with any GUI, or a machine that has not booted yet.

agent ──► plugin ──► ssh ──► node: qm vncproxy ──► unix socket /var/run/qemu-server/<vmid>.vnc
                      └────► node: qm guest exec ──► qemu-guest-agent inside the VM

What you get

• Pixels — console_shot returns the VM screen as an inline image (with region and scale to keep the token cost sane), and mouse/keyboard tools operate in VM screen coordinates: move, click, double click, drag, wheel, type, key combinations, and a batch runner that performs a chain of actions and returns a single frame at the end.
• Commands — vm_exec, vm_file_read, vm_file_write, vm_status through the QEMU guest agent. Deterministic and cheap: use pixels only where a GUI really is in the way.
• Reliable input — uppercase goes through Caps Lock (qm monitor → HMP sendkey), shifted symbols through RFB keysyms with a settle delay. This is what makes typing a password on the Windows login screen work instead of producing xxXX → xxxX.
• Unicode text — a Windows guest gets a scheduled task in the logged-in user's session (SendKeys), a Linux guest gets xdotool/wtype through the guest agent. Layout and modifiers stop mattering.
• Clipboard — console_paste puts UTF-8 text into the guest clipboard and presses Ctrl+V; console_clipboard reads the guest clipboard back, so text can leave the VM without OCR. The guest agent is the primary channel (a scheduled task in the logged-in Windows session, xsel/xclip/ wl-copy/wl-paste on Linux), so it also works on SPICE VMs (vga: qxl) where the VNC clipboard cannot reach the guest. The VNC clipboard (clipboard=vnc) remains as a fallback.
• A settings card in the web UI (English, Russian and Chinese labels), backed by the profile patch layer.
• A live screen panel — the “VM screen” tab in the right sidebar shows the frames the agent sees, taken from the very same VNC session, while the agent works. It follows the agent's VM automatically or stays on the VM you pick by hand. Watching only: the panel never clicks or types.
• Zero runtime dependencies — RFB, DES authentication, keysym tables and PNG encoding are implemented in this package on top of node:crypto and node:zlib. There is nothing to install on the host, and nothing to install on the node either.

Verified against pve-manager 8.4.21 with Windows 11 / Windows Server 2025 and Linux guests, on DeepSeek Harness 0.2.0-rc.2.

Requirements

Install

# 1. Add the bundle
dsh plugin --profile web add github:taxah92/dsh-pve-vm

# 2. Make sure the bundle is enabled in the profile: dsh.profile.bundles must list "dsh-pve-vm"
#    (the plugin manager / market toggle does the same thing)

# 3. Restart the harness:  the host half of a plugin is loaded once per process

The plugin adds one row to the profile patch layer:

- id: pve-vm
  name: dsh-pve-vm

The row id pve-vm is the settings namespace — do not rename it, or the settings card and the patch layer will stop finding each other.

Configure

Settings ▸ Plugins ▸ Proxmox VE in the web UI. Everything is written into the profile patch layer, and the same values can be edited there by hand:

- id: pve-vm
  name: dsh-pve-vm
  config:
    sshHost: proxmox            # alias from ~/.ssh/config, or an address
    sshUser: root
    sshPort: 22
    sshKeyPath: ''              # empty — default keys
    sshConfigFile: ''           # like `ssh -F`
    sshExtraOptions: []         # extra -o options, one per item
    allowVmids: [100, 101]      # VMs the plugin may touch; empty list forbids everything
    enableConsole: true
    enableGuest: true
    inputChannel: auto          # auto | hmp | vnc
    sendkeyDelayMs: 80
    consoleTimeoutMs: 20000
    sessionIdleMs: 300000
    guestTimeoutMs: 60000
    guestTypeCommand: ''        # script for typing non-ASCII text into a Linux guest
    viewAutoOpen: false         # open the "VM screen" tab by itself when the agent touches a console
    viewFollowAgent: true       # the panel follows the VM the agent works with
    viewFps: 2                  # frame pace in the panel, 1–4
    viewScale: 2                # frame resolution divisor: 1 native, 2 half, 4 quarter

First run checklist

1. allowVmids is empty by default — nothing is allowed until you list a VM id. This is the only gate the plugin has.
2. ssh <alias> 'qm status <vmid>' should work from the host without a password prompt.
3. ssh <alias> sshd -T | grep -i acceptenv must include LC_* (see Requirements).
4. Call vm_status — it reports whether the VM is allowed, whether the guest agent answers, which guest OS was detected, and whether a console session is open.

Tools

Console (VNC)

Every action tool accepts a common tail: observe (return a frame after the action, default true), wake (Shift + pointer jitter to wake a blanked display), settleMs, scale, region.

Guest agent

Watch the VM screen live

The VM screen tab in the right sidebar shows what the agent sees while it works. Add it from the sidebar guide, or let viewAutoOpen reveal it the moment the agent touches a console.

• The frames are taken from the agent's own RFB session — no second VNC session appears and the qm vncproxy password is never reset under the agent's feet. Captures and the agent's tools share one session queue, so they never tear the stream for each other.
• viewFollowAgent keeps the panel on the VM the agent is working with; clicking a VM chip pins the panel to that VM instead. viewFps (1–4) and viewScale (1 / 2 / 4) control the pace and the resolution, and both can be changed on the fly inside the panel.
• The panel is watch-only: no clicks, no typing, nothing is injected into the guest. A blank frame is called out explicitly — the guest blanked its display or is locked.
• Frames are served by the plugin's own routes (/pve-vm/console/frame, /state, /release) on the harness web server, behind a loopback-only fence: a request is answered only when the Host is loopback (localhost, 127.0.0.0/8, [::1]), a present Origin matches Host, and sec-fetch-site: cross-site is refused. If you bind the harness to your LAN, the panel stays dark by design — the same fence the harness applies to /api.
• The fence is loopback, not the harness token. These routes live outside /api, so the harness URL token does not cover them (the browser-skill overview works the same way). Any process on the same machine can therefore pull frames or call /release; that is the accepted price of loading frames without a token on a single-user box. Serve VMs you would not mind a local user seeing, and harden with a harness-issued token if that is not acceptable.
• The console session stays open while the panel streams (the idle reaper does not take it); the panel's “disconnect console” button closes it immediately. Streaming pauses by itself when the tab is hidden.

How it works

• Transport. A VM has no TCP VNC port: QEMU listens on the unix socket /var/run/qemu-server/<vmid>.vnc, and qm vncproxy bridges it to its own stdin/stdout. The plugin therefore speaks RFB through an ssh child process — no ports, no TLS, no websocket, and the VNC password never lands in argv. qm vncproxy treats EOF on its stdin as "quit", so the plugin keeps that pipe open (the bridge also drops idle sessions after 15 minutes; the plugin closes its own session after sessionIdleMs, 5 minutes by default).
• Input channels. inputChannel: auto (default) mixes them: Caps Lock through the QEMU monitor for uppercase, RFB keysyms for the rest. hmp sends everything through qm monitor sendkey (faster, but QEMU occasionally loses modifiers below ~80 ms), vnc uses RFB only (uppercase is unreliable — QEMU emulates Shift itself and our own Shift events desynchronize it).
• Unicode. Windows: the plugin drops a payload file and a PowerShell typer into the guest and registers a scheduled task with -LogonType Interactive, so the text is typed inside the user's session via SendKeys — layout and modifiers stop mattering, and the VM is not restarted. Linux: xdotool type (X11) or wtype (Wayland) inside the active session; both paths run through the guest agent in a single call.
• Guest channel. All qm calls for one VM (guest exec, status, OS info, sendkey through the monitor) are serialised with a 250 ms gap: QEMU serves a single QMP client at a time and PVE's QMP timeout is 5 s, so back-to-back calls can steal each other's replies. A failed call is retried once, and only when PVE answers “agent is not running” — that check happens before the command is sent, so the retry cannot run anything twice. A file chunk carries the request marker and the real file length next to the data, so a foreign reply (a reused pid) or a short one is retried instead of being mistaken for the end of the file.
• Output encoding. Guest bytes are decoded from the raw bytes: UTF-8 when they are valid, otherwise the console OEM code page (chcp) or the system ANSI code page — whichever reads as text. Windows mixes the two: cmd.exe messages arrive in ANSI even when chcp reports OEM. Non-ASCII command text travels as a base64 envelope instead of the raw QMP request, and file data goes over stdin: both work around byte-per-character truncation in PVE (#6609) and the ~32 KB command-line limit of a Windows guest.
• Clipboard. Two channels. The guest-agent channel runs a scheduled task in the logged-in Windows session (Set-Clipboard/Get-Clipboard; the clipboard is per-session, so SYSTEM in session 0 cannot see it) or xsel/xclip/wl-copy/wl-paste in the active Linux session; it works on SPICE VMs (vga: qxl) and needs a logged-in session. The VNC channel uses ClientCutText for writing and the extended clipboard pseudo-encoding (VNC_ENCODING_CLIPBOARD_EXT) with CAPS/REQUEST/PROVIDE for reading; QEMU passes those bytes to the guest through qemu-vdagent, so that VM needs vga: <type>,clipboard=vnc and a full stop/start (device arguments are fixed at start-up), plus a SPICE agent in the guest. It is used as a fallback when the guest-agent channel is unavailable.
• Frames. RFB Raw/CopyRect rectangles are composited into RGBA (QEMU leaves alpha at zero), optionally cropped and downscaled, then encoded to PNG with node:zlib.

Troubleshooting

Security

This plugin gives an agent root-level hands inside your VMs. Be deliberate about it:

• Every action is executed without an approval prompt. The only gate is allowVmids, which is empty by default; list only the VMs you are willing to let an agent touch, and keep the list short.
• vm_exec runs as root/SYSTEM in the guest, vm_file_write can write anywhere the agent can, and the console tools can drive a logged-in desktop. Treat an allowed VM as a compromised host: do not keep long-lived secrets in it while the plugin is enabled.
• Root SSH to the node is required (qm needs it). Use a dedicated key, restrict it to the nodes you need, and consider from=/command= restrictions in authorized_keys where possible. The plugin passes the whole host environment to the ssh child, as any ssh call does.
• sshExtraOptions accepts arbitrary -o options, including ProxyCommand — it is a privileged escape hatch for the person who configures the plugin, not for the model.
• VNC passwords are generated per session (8 printable ASCII characters) and handed to qm through the environment via SendEnv, so they do not show up in ps. The QEMU password is scoped to the VM and expires with the session.
• The plugin talks only to your node. No telemetry, no outbound connections of its own.

Limitations

• Proxmox VE only, QEMU VMs only — no LXC containers, no other hypervisors.
• Clipboard tools move text only (no files, no images) and only inside a logged-in session: the guest-agent channel needs a running qemu-guest-agent and a desktop session, the VNC fallback additionally needs clipboard=vnc on the VGA device and a SPICE agent in the guest.
• Unicode typing in Windows needs an interactive session; on the login screen only ASCII through console_type works.
• One console session per VM id, serialized with a mutex; the session is closed on idle.
• The console tools do not work with a serial console (vga: serialN) — there is no framebuffer.
• Not yet verified: Proxmox VE 7 and 9, multi-node clusters, Linux guests with Wayland-only desktops.

Development

npm run check                       # node --check on every runtime file (no harness needed)
node tools/check-client.mjs         # browser bundle format (classic script, exports, inject)
node tools/check-plugin.mjs         # settings schema, tool set, JSON schemas, layout contract

check-plugin.mjs imports the plugin, and the plugin imports the harness packages (@deepseek-ai/dsh-tools, @deepseek-ai/schemastery), so run it where those resolve — the easiest way is to install the checkout into a profile with link: and run the script from there:

dsh plugin --profile web add link:/path/to/dsh-pve-vm
cd /path/to/dsh-pve-vm && node tools/check-plugin.mjs

For a bare checkout, point DSH_PACKAGES at the harness package directory (…/@deepseek-ai/dsh/node_modules/@deepseek-ai) — the script uses it as a fallback for the compiler it needs.

Live probes need node access only (no harness):

node tools/probe-rfb.mjs --vmid 100 --ssh-config ~/.ssh/config --out /tmp/shot.png
node tools/probe-rfb.mjs --vmid 100 --click 640,400 --after /tmp/after.png
node tools/live-test.mjs --vmid 100 --ssh-config ~/.ssh/config --out probe-out
node tools/clipboard-test.mjs --vmid 100 --text "hello" --out /tmp/clipboard.png

All of them talk to the node exactly like the plugin does (qm vncproxy over ssh), so they are the fastest way to tell a plugin bug from a node-side problem.

License

MIT — see LICENSE.

Proxmox® is a registered trademark of Proxmox Server Solutions GmbH. This project is an independent plugin and is not affiliated with or endorsed by Proxmox Server Solutions GmbH or by the DeepSeek Harness project.