dsh-terminal-vision

dsh-terminal-vision

Terminal semantic vision plugin for DeepSeek Harness (DSH).

Gives the agent "semantic vision" over interactive terminal programs — vim, htop, lazygit, less, setup TUIs — instead of a raw escape-sequence byte stream. Every PTY session is continuously parsed by @xterm/headless (the headless JS port of xterm.js) into a structured screen model: visible text lines, cursor position, active line. The agent can then read the screen, send keys, read again — a perception/action loop over any TUI.

Tools registered in DSH

Architecture

• Host part (TypeScript, src/) — Cordis plugin: apply(ctx) registers the tools on ctx.tools, manages persistent sessions via node-pty (prebuilt ConPTY binaries on Windows — no compiler needed).
• Semantic parser (src/screen-model.ts) — Variant A: @xterm/headless builds the screen model natively in Node. Raw PTY output is written into a headless xterm Terminal; the buffer is read out on demand (buffer.active, translateToString). No Python, no long-lived bridge process, works offline.
• Keys are translated by encodeKeys() (friendly names → terminal bytes, including Ctrl/Alt modifiers, arrows, F-keys).
• Readiness wait: after each terminal_send_keys, the tool waits until the screen revision changes (data arrives) or the timeout elapses — preventing "blind" sends.
• All tools return JSON-serializable objects; unknown session ids return a clear error instead of crashing.

Install

# from a local checkout
dsh plugin --profile web add file:./dsh-terminal-vision

# or from GitHub
dsh plugin --profile web add github:fdfdfdfgbfdgfdg-star/dsh-terminal-vision

Windows note: pnpm blocks native-module install scripts by default. If the add step fails with ERR_PNPM_IGNORED_BUILDS: node-pty, run the blocked build step manually (node-pty ships prebuilt ConPTY binaries — no compiler needed):

cd ~/.dsh/profiles/web
pnpm rebuild node-pty     # runs the postinstall that copies conpty.dll

Then restart DSH (relaunch the DeepSeek Harness app, or re-run dsh --profile web) and check the boot log for:

[terminal-vision] loaded: tools=6

Verify in a chat: "Какие инструменты терминала тебе доступны?" — the agent should list the six terminal_* tools.

The package declares a dsh.bundle.patch (cordis.patch.yml), so dsh plugin add wires the plugin into the profile automatically — no manual cordis.patch.yml edits needed.

Example session

terminal_open(session_name="htop", shell="htop")     → session_id
terminal_get_screen(session_id)                      → lines[] of the htop interface, cursor
terminal_send_keys(session_id, "F10")                → quit htop, new screen
terminal_close(session_id)

Driving a shell:

terminal_open(session_name="work", shell="bash")
terminal_send_keys(session_id, "ls -la\n")
terminal_get_screen(session_id)                      → the ls output
terminal_read_output(session_id, lines=200)          → anything that scrolled off

Build & test (development)

npm install
npm run build      # tsc → lib/
npm test           # node --test test/integration.test.js

lib/ (compiled JS) is committed, so a plain dsh plugin add needs no build step. For the same reason the package deliberately has no prepare script: pnpm blocks git-hosted packages from running build scripts on install (ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED), and since lib/ is already in the repository the build is unnecessary. When developing, run npm run build and commit the updated lib/.

Dependencies

• Node.js >= 22.19 (DSH host requirement)
• node-pty ^1.1.0 — ships prebuilt Windows ConPTY binaries; no Visual Studio Build Tools required on Windows 11
• @xterm/headless ^6.0.0 — pure JS, no native code
• No Python, no root/admin rights

Known limitations

• The screen model is text-only: colors/attributes are parsed by the emulator but not exposed in the snapshot.
• cursorHidden is advisory (xterm.js does not expose the DECTCEM state); most TUIs keep the cursor visible.
• Alt+<letter> sends ESC-prefixed bytes (application keypad mode not negotiated per-program).
• Sessions live in the DSH host process; restarting DSH closes all sessions.
• On Windows the default shell is cmd.exe (ConPTY); bash works if Git Bash / WSL bash is on PATH.