JeremyWangCY/dsh-pc-pilot

dsh-pc-pilot

English | 中文

A DeepSeek Harness host plugin that gives the model a single computer tool to observe and operate the local Windows desktop: an indexed UIA accessibility tree, per-window screenshots, background synthetic-cursor input that never steals focus, and — when a task truly requires it — real SendInput mouse/keyboard control.

While acting, the model moves a small on-screen cursor indicator (a rounded arrow with a soft blue radial glow) to each target point, and a frosted-glass status pill at the top center of the screen shows that PC-Pilot is running (with a breathing green dot on the right). Both are click-through, never take focus, and auto-hide three seconds after the last action.

Runtime scope

PC-Pilot targets one interactive Windows session: the user keeps working normally while the AI prefers background-safe paths. Browser CDP, UIA patterns, target-window messages, WGC and PrintWindow are used to avoid focus theft whenever the application supports them. Virtual machines, a second desktop, or a hidden alternate Windows session are not part of the architecture. If an application truly requires real foreground SendInput, PC-Pilot reports that its background path is unavailable rather than pretending independent user/AI input can be guaranteed.

Optimization priority: browser batched/local observation → UIA caching and capability routing → reusable WGC capture sessions → conditional batching. Capture support and background-input support are evaluated separately: being able to capture a covered window does not imply that the same application can accept safe background input. Application-specific object models, automation APIs, and domain integrations belong in separate plugins rather than PC-Pilot core.

Features

• One tool, full desktop plus browser — the desktop surface uses canonical Windows Computer Use names directly: list_apps, list_windows, get_window, launch_app, get_window_state, click, press_key, type_text, scroll, drag, set_value, perform_secondary_action, activate_window, and minimize_window.
• Reusable Windows targets — observations return window: { id, app }, which can be supplied unchanged to the next action. Element actions use element_index; screenshot coordinates use scrollX / scrollY and mouse_button where applicable.
• Background-first input with capability routing — observations cache per-window/per-control support for UIA patterns (Invoke / Value / Toggle / Selection / ExpandCollapse / Scroll / RangeValue). Known-unsupported paths are skipped instead of reprobed on every action; verified target-window WM_CHAR / WM_KEY / WM_MOUSEWHEEL messages cover traditional controls without consulting an occluding foreground window. A fresh UIA observation rebuilds the capability map when the window/control tree changes.
• State-bound actions — get_window_state returns a snapshot_id and screenshot_id; element actions must present that snapshot and coordinate actions can bind to the screenshot. Expired, moved, wrong-window, changed-element, or consumed state is rejected instead of falling back to a potentially wrong control.
• Persistent browser-use session — one bounded CDP WebSocket is reused per isolated AI browser endpoint, with reusable per-tab sessions rather than reconnecting for every action. browser_tabs, browser_history, browser_back / browser_forward, and condition-based browser_wait provide session-level navigation; browser_events returns cursor-based console/network/lifecycle evidence and browser_downloads tracks Chromium download progress plus completed files. Screenshot observation has its own non-fatal timeout so a slow frame cannot tear down an otherwise healthy browser session. browser_state preserves raw tokens plus compact @eN refs for compatibility, while browser_observe returns the compact refs without UUID token noise; both remain bound to the exact tab/document/name/role. browser_click, browser_type, browser_replace, and browser_key accept either form. launch_app { app: "msedge.exe", headless: true } still starts a fresh isolated profile and browser_* never attaches to the user's own browser.
• Stable UIA identity + incremental state — include_text: true assigns every returned control a stable element_id, plus monotonic accessibility_revision and accessibility_delta metadata (added, removed, changed, unchanged_count). Existing snapshot-bound element_index actions remain fully compatible; the stable identity is for reasoning across observations, not for bypassing stale-snapshot checks.
• Verified postconditions and deterministic recovery — actions can carry an expect contract for window presence/closure, accessibility change, stable-element value, desktop text, browser URL/text/readiness, or completed downloads. A failed postcondition is reported as postcondition_failed and is never blindly replayed. recovery: "foreground_once" is intentionally narrow: it may retry only after an explicit not_executed + background_unavailable; element recovery requires the observed stable element_id so PC-Pilot can re-observe and remap a fresh index/snapshot first.
• Stable app identity and exact targeting — list_apps, list_windows, and get_window expose an app_identity with Win32 executable identity or packaged-app AUMID, parent pid, and same-process-family root pid. Reuse identity_key to target a known app exactly instead of relying on title/process-name matching. get_app_identity lazily adds product/version/company metadata and optionally verifies Authenticode signer publisher/subject/thumbprint with executable-level caching.
• Bounded failure semantics — one-shot helper calls have an external deadline watchdog. A timeout, disconnect, or post-dispatch transport error reports outcome: "unknown"; mutating actions are never replayed automatically.
• Lightweight conditional batching — send up to 20 ordered actions through actions. Any step may use when as a precondition and expect as a verified postcondition; the next step runs only after the current gate is verified. when checks current state once by default (timeout_ms: 0) and never dispatches the action when unmet. There is deliberately no branching/workflow DSL or hidden replay engine; execution still stops at the first failed, unmet, or uncertain step.
• Structured safety classification — obvious consequential target names and sensitive browser fields are classified in the result for future host policy integration; the current PC-Pilot profile does not interpose confirmation.
• Occlusion-immune background clicks — with an app specified, coordinate clicks aim at the target window's own UIA tree / hwnd, so a fully covered window can be operated unattended while the user keeps working on top.
• Rich mouse vocabulary — click supports standard left, right, wheel (middle), back, and forward buttons plus a bounded click_count; scroll preserves simultaneous scrollX / scrollY axes; drag uses path.
• Computer-use action spelling compatibility — double_click, type, keypress { keys: [...] }, and move are accepted alongside the Windows canonical click, type_text, press_key, and mouse_move actions.
• O(1) element lookup — get_window_state caches the UIA element list inside the persistent helper daemon, so click { element_index }, set_value, perform_secondary_action, select_text, and type_text resolve without a second full-tree traversal.
• Occlusion-immune screenshots with reusable WGC sessions — a bundled Windows Graphics Capture bridge captures the target HWND even when it is covered and reuses a bounded GraphicsCaptureItem + FramePool + CaptureSession slot per HWND. Window-size changes recreate only the frame pool; closed, failed, or idle slots are evicted. PrintWindow (multi-flag ladder) remains the occlusion-immune fallback. Neither tier copies the visible screen, so a covering window can never leak into the shot; when neither can render, the result is a legible screenshot_black error instead of a frame of the occluder. The bridge activates when .NET 8 is available; PrintWindow keeps the plugin usable without it.
• Per-task dispatch — dispatch: "foreground" (real SendInput) exists for the cases that genuinely need it (canvas clicks, unsupported drags, apps with no background path); the tool guidance keeps background as the default and asks the model to be explicit when it goes foreground.
• Virtual-cursor indicator — per-pixel-alpha layered window (UpdateLayeredWindow + CreateDIBSection): rounded white arrow with black outline over a soft blue radial glow. Click-through (WS_EX_TRANSPARENT), non-activating (WS_EX_NOACTIVATE + SW_SHOWNOACTIVATE), always-on-top. Auto-hides 3 s after the last action and reappears on the next one.
• High-DPI accurate — the overlay calls SetProcessDPIAware at startup and positions itself in physical pixels, matching the physical coordinates the helper reports from UIA. Correct placement at 100% / 125% / 150% scaling.
• PowerShell 5.1 + 7 (Core) compatible — the overlay adds the System.Private.Windows.GdiPlus / System.Private.Windows.Core references under .NET Core so both runtimes render the cursor identically.
• Frosted status pill — while the overlay is active, a dark frosted-glass pill sits top-center on the primary screen reading "PC-Pilot 运行中" with a breathing green dot on the right (Apple-style: rounded capsule, subtle top sheen, sine-wave breathing). Same per-pixel-alpha layered-window technique as the cursor, TOPMOST + NOACTIVATE + click-through, so it never intercepts input; it hides 4 s after the last activity.

Requirements

• Windows 10 or 11
• DeepSeek Harness (DSH) with the dsh-pc-pilot bundle loaded in the web profile
• PowerShell 5.1 (built into Windows) and Node.js ≥ 22.12 (shipped with DSH)
• .NET 8 Desktop/Runtime is optional; when present, the bundled WGC bridge provides occlusion-independent native HWND screenshots. Without it, occlusion-immune PrintWindow capture keeps the plugin usable.

Installation

Published on npm. npm itself is included with Node.js.

From the DSH plugin market

Once listed, search for dsh-pc-pilot in the market and click install.

From npm

npm install dsh-pc-pilot

Run this inside the DSH profile (~/.dsh/profiles/web), then restart the host.

From source

git clone https://github.com/JeremyWangCY/dsh-pc-pilot.git
cd dsh-pc-pilot
npm install ./dsh-pc-pilot

Or link it manually: add "dsh-pc-pilot": "link:./vendor/dsh-pc-pilot" to the profile's package.json dependencies, add the bundle to dsh.profile.bundles, run npm install, and restart the host.

CLI and runtime API

PC-Pilot also exposes a small standalone runtime surface. The CLI is intentionally thin: it does not maintain an action whitelist, invent workflows, or retry uncertain mutations for the agent.

npm install -g dsh-pc-pilot
pc-pilot status
pc-pilot doctor
pc-pilot list_apps --json
'{"action":"list_apps"}' | pc-pilot request --stdin --json

act / direct action mode is a convenience layer. request is the escape hatch for agents: it forwards a complete computer request, including batches and future fields, to the same core used by the DSH computer tool.

import { createPcPilotRuntime } from 'dsh-pc-pilot/runtime'

const pc = createPcPilotRuntime()
const apps = await pc.act('list_apps')
const batch = await pc.run({ actions: [{ action: 'wait', seconds: 1 }] })

// Optional convenience only: bind a reusable target without creating a hidden session.
const launched = await pc.act('launch_app', { name: 'msedge.exe', headless: true })
const opened = await pc.act('browser_open', { browser: launched.browser, url: 'https://example.com' })
const page = pc.bind({ browser: opened.browser })
const observed = await page.act('browser_observe')
await page.act('browser_click', { browser_element: observed.elements[0].ref })

pc.close()

The runtime returns the core outcome unchanged. In particular, outcome: "unknown" is evidence for the agent to inspect state; the CLI does not turn it into an automatic replay. runtime.bind(defaults) is only a shallow request convenience: per-call fields win, nested window / browser targets merge, and no session daemon, lifecycle, page choice, observation, navigation, or retry is added behind the agent's back.

The browser backend is also a small injectable provider rather than a second policy layer. The default provider keeps the current persistent CDP session implementation; alternate providers only need an execute(action, args, signal) function. This leaves room for a BrowserSkill-compatible backend without forcing its workflow onto every agent.

Status pill

While the overlay is enabled (default), each first action launches two tiny resident PowerShell loops from the helper directory: the virtual-cursor indicator and the frosted status pill. They read a state file under %TEMP%\dsh-cua — the pill is visible top-center while activity is fresh (≤ 4 s) and fades out afterwards; both processes idle-exit after 120 s and respawn on demand. No driver, no UAC, no display changes — PC-Pilot runs entirely on the user's real desktop, in the background.

Usage

The plugin registers one global tool, computer. Typical flow:

1. computer { action: "list_apps" } — running apps with pids, exact app_identity / identity_key, window titles, hwnds and rects.
2. computer { action: "get_window_state", window: { id, app }, include_screenshot: true, include_text: true } — indexed accessibility tree with stable element_id, revision/delta metadata, a window screenshot and snapshot_id.
3. Act on the state — element actions include the snapshot_id from the same observation. Browser results return a reusable browser: { endpoint, tab_id } target that can be passed back unchanged; element actions can use either the raw browser_element token or the short @eN ref returned by browser_state / browser_observe.
4. Observe again only when state is stale/unknown, the target changed, or the next decision needs information you do not already have. Desktop element_index values remain bound to the get_window_state that produced them.

Action reference (57 actions)

Key parameters

How it works

model ── computer tool ──> host (Node ESM bundle)
                              │  persistent PowerShell daemon, bounded one-shot fallback (JSON in / JSON out)
                              ├─> UIA accessibility tree (IUIAutomation)
                              ├─> Windows Graphics Capture → PrintWindow screenshots (occlusion-immune, no screen-DC tier)
                              ├─> background input: UIA patterns → pixel hit-test → WM_* messages
                              ├─> foreground input: SendInput
                              └─> overlay: UpdateLayeredWindow per-pixel-alpha layered window

The helper is a single self-contained pc-pilot-helper.ps1 copied to %TEMP% once per host start; the overlay is virtual-cursor-overlay.ps1 and the status pill is pcpilot-statusbar.ps1 — resident low-frequency loops that read state files and blit per-pixel-alpha layered windows with UpdateLayeredWindow. Registration uses the harness tool API (defineTool + ctx.tools.register) with isConcurrencySafe: false, so desktop actions serialize.

Security considerations

• The tool can read window titles, accessibility trees and screenshots, and can drive input into the user's applications. The bundled tool description instructs the model to operate only what the user explicitly asked for and to never submit forms, send messages, make purchases, delete data, or change account/settings without explicit instruction.
• Background actions never move the user's cursor or steal focus. Foreground actions do — the guidance requires the model to say so.
• No telemetry. The plugin performs no network access and no persistence beyond `%TEMP%\dsh-cua