dsh-browser-use

English · 中文

@ricardochen1996/dsh-browser-use

Give DeepSeek Harness a browser that can *see and click*: a page is read as an indexed action-space table, and the model does exactly one operation on one observed target per step.

This is a DSH plugin (a host half + a web half). It contains no browser logic of its own — it hands requests to the Jev Ultrafast engine, so there is only one loop implementation. The engine ships inside the npm package as a pinned wheel, and the plugin installs it into its own Python environment the first time it loads, with a Python 3.12+ the machine already has (DSH ships one) or with uv when it has none: no engine clone, no manual Python setup, nothing to install by hand.

⚡ The point: speed

The focus is fast browser decisions. Jev picks an operation and target from the current action table; a text model supplies values only when needed for TYPE_TEXT. browser_goal runs the decision loop inside the engine. browser_act({ observation, intent }) delegates one decision to Jev — or, with jev.decider: chat, to the text model — but still returns to its calling agent after each step. Jev is always available; it does not automatically replace every chat-model turn.

The same local hotel task (type a city, tick two filters, search, open a result), running jev and the "current chat model" in alternating arms, 3 runs each, every run independently verified:

Mode · How each step is decided · Whole task (median) · Relative Jev on ⚡ · one jev-1.13 choice · 10.7 s · baseline Jev off (reasoning low) · a full deepseek-v4.1-flash turn · 24.6 s · 2.3× slower Jev off (reasoning max, current DSH setting) · a full deepseek-v4.1-flash turn · 26.7 s · 2.5× slower

In this decision-loop benchmark, Jev took about 60% less time than the max-reasoning arm (26.7 s / 10.7 s ≈ 2.5). Median per-step decision latency was 1.1 s versus 2.8–3.6 s.

Measured on 2026-09-30: three arms, three completed runs each, all independently verified; one transport failure was recorded and retried. The clock runs from the first decision after page opening to DONE/finish. The script directly loops over the sidecar's intent path versus lean Responses API tool-calling turns using deepseek-v4.1-flash, the model configured in DSH at measurement time. This is not an end-to-end DSH toggle comparison: it excludes browser startup, initial navigation, post-run verification, DSH delegation and outer chat round-trips. Three runs of one local task do not establish a universal speedup. The Jev text helper uses the same model with thinking disabled; session mode shares routing settings, not necessarily protocol or reasoning settings. See bench/ for the script and raw data. The benchmark spends model quota and reads credentials only from environment variables.

Repository layout

dsh-browser-use/
├── lib/          Node half: index.js (assembly + delegation) / tools.js (tools & delegate tool) /
│                 sessions.js (browser ownership, attach exclusivity) / sidecar.js (process, cancel, generation) /
│                 engine.js (interpreter & environment probing) / provision.js + python.js (installs the engine) /
│                 inspector.js + client.js (side panel)
├── bin/          doctor (self-check, --install) / vendor_engine.py (rebuild & verify the bundled engine) /
│                 vendor_requirements.py (render & verify vendor/requirements.txt) / release checks
├── sidecar/      Python half: bridge.py — the only place that imports the engine
├── vendor/       the engine wheel the plugin installs, the hash-pinned requirements.txt pip installs,
│                 and jev-ultrafast.json (the engine revision the wheel was built from)
├── test/         Node checks (plugin.mjs / delegation.mjs / inspector.mjs / provision.mjs) + Python checks (pytest + two e2e)
├── pyproject.toml / uv.lock   the sidecar's Python environment (engine from the vendored wheel)
└── package.json  the DSH plugin manifest (dsh.bundle / dsh.client)

The engine (jev_ultrafast) is a separate repository. This package ships one wheel built from a pinned revision of it (vendor/, named in vendor/jev-ultrafast.json) and none of its source.

1. What it provides

Browser operations are not in the main agent's hands. The conversation sees only these tools (browser_task_status exists only with a persistent subagent):

Tool · Job browser_task · Hand a goal to this conversation's persistent browser subagent: the first call starts it, and every later call sends the new task to the same subagent (it remembers the page and what it did before). The call returns immediately; short progress pushes arrive while it works and the conclusion comes back later as a message. fresh: true swaps in a new subagent with no memory. browser_task_status · A snapshot on request of what the persistent browser subagent is doing: working (for how long, on which task, on which page) or idle (how its last run ended, its closing message). You do not need it to watch progress — progress is pushed to you and the conclusion arrives as a message anyway — so `wait: true` is the last resort: it blocks until the subagent finishes or messages you, or the user writes (timeout_ms, default 2 min, at most 10 min), and is only for a step that cannot go on without the result. browser_doctor · Self-check: interpreter, engine version, engine install progress, browser, delegation status (including whether this conversation's subagent is working), and a fix command for each missing piece. install: true installs the engine now, or retries a failed install.

The browser tools live in the delegated subagent's scope (7 by default, with browser_goal off). Only it can see and call them:

Tool · Job browser_open · Open a page in the browser this Session owns and return the action-space table. browser_page · Re-observe and produce a new observation number. browser_act · Run one operation on one target from an observation: CLICK / TYPE_TEXT / SELECT / SCROLL_UP / SCROLL_DOWN / WAIT. With jev on you may instead pass an intent: with no operation, the policy picks the operation and target on that observation; a TYPE_TEXT with no text gets its value from the text model. browser_eval · Evaluate one JavaScript expression in the page and return its value — a string as itself, anything else as JSON, capped. It is the one tool that takes code, and it runs on the Session's own CDP connection so a model that needs a programmatic read never starts a second client. It produces no observation: read with browser_page before acting. Withdrawn by allowEval: false. browser_screenshot · Capture the viewport (image attachment). browser_console · Read console messages, page exceptions, failed requests and 4xx/5xx. browser_goal · Hand the whole goal to the browser policy in one run: it picks every operation and target itself (spends paid quota; jev.decider says whether Jev or the text model answers). browser_close · Close the tab and the daemon. In mode: attach the attached browser itself stays open, whether or not other Sessions are sharing it.

One connection, one driver. Every tool above — browser_eval included — works over the single CDP connection this Session's daemon holds. A second client (a raw websocket to Chrome's DevTools port, a puppeteer/playwright script, another browser-harness daemon) is a second connection, and Chrome 144+ asks for remote debugging once per connection: each client raises its own "Allow remote debugging?" sheet, and an unanswered one is only kept on screen by the client that raised it — dropping that client and retrying raises another. That is how one approval turns into a stack of prompts. The plugin never does this; a browser agent must not either, which is why the rule is in its prompt and browser_doctor repeats it in mode: attach. If the prompt does appear, approve that exact sheet once and never retry.

Keep delegating without opening a pile of `browser_task`s. The plugin uses DSH's native continuable subagent (ctx.subagents.startContinuable / sendMessage); each conversation has exactly one browser subagent:

• New task: call browser_task again — the task is delivered as a message into the same subagent's conversation; if it is busy it is admitted at the next step boundary, if idle it starts right away, and if it was released it cold-starts from persistence (browser tools re-attach automatically).
• Two-way messaging: the subagent is granted only the global send_message, so it can send you progress, questions and results at any time; you use send_message to add to or correct the current task, and interrupt_agent to stop it.
• Progress while it works: the plugin pushes its own record of each browser step (Sessions.observed() / noteAction()) into this conversation as an agent-message from the subagent, coalesced to at most one per `progressIntervalMs` (default 30 s). A push wakes a conversation that has ended its turn, so you end your turn or keep working and the next one wakes you — no polling and no blocking, and wait: true is not how you watch progress. Every push is two lines:

Browser subagent "<childId>" progress — <duration> into the task (status only; no reply needed).
  Page: <title> — <url> (observation 1).

That is a first push: it comes right after the page is read, so no action has run yet. The , last action: clause joins in once an action has run, and — appears only when the page has a title. Before the first page read the second line is It has not read a page yet. A push is informational only: do not reply to it, do not re-task, and do not treat it as the report — the report is the conclusion below.

• Results come back on their own: the subagent sends its conclusion back; when it goes idle DSH also sends the main agent a "Background subagent … finished" notification and wakes it, so neither polling nor blocking is ever needed. The browser_task receipt says so explicitly — keep working or end the turn, never sleep in a shell — and, where an agent team is present, that its wait_agent / list_agents do not see this subagent.
• Status on request: browser_task_status reads the subagent's state from the host's own subagent/start / subagent/end events and the agent registry. With wait: true it blocks on the same settlement the notification rides on, instead of guessing with sleep, and returns early when the user writes or the subagent messages the main agent — reserve it for a step that cannot go on without the result.
• Survives restarts: the subagent is recorded under the tag browser-task in the parent session's directory, so after a DSH restart or resume the next browser_task / send_message finds the same subagent.

When the provider doesn't support continuable subagents (or delegateMode: one-shot is set), it falls back to one-shot delegation: one subagent per task, and browser_task returns after it reports back.

When the composition has no usable subagent provider (or delegate: false), the plugin falls back to direct mode: the browser tools are mounted on the composition and called from the chat itself — the same implementation, just without a subagent.

Here is what the subagent sees (real output):

Page https://www.baidu.com/ — 百度一下，你就知道
Observation 3. Targets below are valid only for this observation.
[13] textbox 国台办回应鲁比奥涉台言论 — TYPE_TEXT, CLICK
[14] button 百度一下 — CLICK
Without a target: WAIT

The side panel also has a Browser agent tab: URL, live screenshot, element table, last action — showing the exact page the tools are driving.

2. Advantages

1. Targets come from the observation, not from the model's imagination

The decision loop outputs (operation, index), never a selector, coordinate or JavaScript. By contrast, Playwright MCP / Chrome DevTools MCP make the model write its own selectors, which miss or misfire the moment the page changes and cannot be validated ahead of time. Here the index is valid only for "the frame it just read". browser_eval is the one deliberate exception (section 1): a separate inspection channel whose output is data, never a decision — it produces no observation and nothing it returns is executed on the plugin's behalf; allowEval: false removes it.

2. Freshness is a first-class citizen

Every decision is bound to the observation fingerprint: if the page changed, it refuses to execute and hands back the new table instead of clicking blindly. Measured on Baidu's homepage: the hot-search list auto-rotates → the first input is intercepted ("Nothing was executed") → re-pick on the new observation → success. No change is ever auto-retried, so there are no double clicks or double submits.

3. Cheap on tokens

Screenshots are not fed to the model (they're for humans); each step sends only structured text: visible text + element table + the operations available this round.

3b. The main agent never reads the page

The table, indices, refusals and re-reads all happen in the subagent's context; the conversation receives only short progress lines and a conclusion. However complex the page, the main agent's context stays the same length — and it has no tool that could misclick a page.

4. The model doesn't have to guess what's possible

The table lists the operations and targets allowed on this frame, including every option of a dropdown (6:1 → Stay category → Design). Unsupported operations, occluded controls and disabled fields simply never appear in the table.

5. One loop, not two

The browser loop, freshness guards and executor all live in the Python engine, covered by 21 real-browser guard checks + 10 sidecar e2e checks + 92 offline unit tests; the plugin is a thin adapter (start the sidecar, render the table, forward signals). Fix one place, both sides agree.

6. Session-level isolation and reclaim

Each DSH Session gets its own browser (its own profile, debug port and daemon) with serialized operations; everything closes when the Session ends. Parallel sessions never fight over a tab or leak state.

The browser belongs to the Session that started the task, not to the subagent: the subagent may be released, restored, or even swapped with fresh: true, and the browser, logins and tabs stay. So consecutive browser_tasks continue on the same page rather than reopening each time. An external browser adopted via mode: attach is held by only one live Session at a time — a second session is explicitly refused. Two Sessions fighting over one external browser is the only real data risk at this layer.

When one browser is meant to be driven by several conversations at once — your everyday Chrome, say — set attachSharing to shared: the exclusivity check stops refusing the second Session, and every Session still gets its own daemon and its own tab. Closing one Session's browser closes that tab and stops that daemon; it never closes the external browser. The default stays exclusive, the "a second session is refused" behaviour above.

One shared-browser detail is worth knowing: when a Session opens its first page, the engine tidies away the browser's unused blank tabs, and a Session that has not opened anything yet is exactly that. Only blank pages are closed, and the affected daemon opens a fresh tab on its next call — so the cost is at most one call that fails on a blank page, not a lost page.

7. It doesn't touch the browser you're using

By default the plugin launches a dedicated Chrome instance. It never touches your personal Chrome profile, so there's no Chrome 144+ "allow remote debugging?" prompt and none of your logins get swept into automation. When you do want to use your own browser, switch explicitly to mode: attach:

• Attach to the Chrome you're using (no `cdpEndpoint`): open chrome://inspect/#remote-debugging in that Chrome and allow remote debugging; the plugin finds it through the DevToolsActivePort Chrome writes into its profile, opens a new tab there to work in, and your logins are available. On macOS this file is privacy-protected, so the app running DSH needs Full Disk Access (System Settings → Privacy & Security → Full Disk Access, then restart DSH), otherwise it reports no_permission clearly. Each DSH Session's first connection triggers Chrome's one "Allow remote debugging?" prompt; click allow. The plugin asks per connection, not per tool: every tool — browser_eval included — reuses that connection, and a browser_close followed by a new browser_open starts a new daemon, so attach mode asks once again on that next connection. Do not open your own CDP client (a raw websocket, a puppeteer/playwright script, another harness daemon) against the same Chrome: Chrome 144+ approves each connection separately, so a second client is a second prompt, and a retry while a sheet is unanswered raises another instead of replacing it. The plugin only closes tabs it opened and never closes your browser at the end. The engine uses exactly the tab the connection layer built for this run — no extra blank tab — and blank tabs you opened yourself are not closed.
• Attach to a dedicated debug Chrome (with `cdpEndpoint`): http://127.0.0.1:9333 or ws://…/devtools/browser/…, see the config table below.

Attach mode is checked when the profile loads, not on the first browser call: the endpoint is asked for /json/version (a ws:// address is asked whether anything is listening), and the Chrome you browse with is looked for the way the engine looks for it. browser_doctor reports Status: ready only when that check passes, and the load log carries the fix when it does not — node bin/doctor.mjs --mode attach asks the same question from a terminal. The check reads only what Chrome already published, so it opens no page and waits for no prompt.

Each Session uses its own profile: if a browser is already running for the same profile, Chrome hands the new launch off to the existing instance and exits (status 0), so the two Sessions share one browser. That's why the plugin partitions by Session; if the host restarts while the browser is still up, the plugin adopts that instance (same profile of the same Session) and keeps its logins and tabs rather than starting another.

A launch leaves exactly one tab: the browser opens a startup tab, and the connection layer opens a separate tab for this run (named daemons can't share one), which the run drives; once the page is ready the startup tab is closed. Cleanup is conservative — it won't close when only a blank page remains, and it never closes a page the user is reading.

8. Debugging is in the tools, not in words

browser_console gives console exceptions and failed requests directly; the panel shows the same page live. Observation and action share one state — there is no "what the tool says" vs "the actual page".

9. Zero build

The host half is plain ESM; the web half is a hand-written __ModuleLoader__ script — no tsdown/rollup step. Clone it, pnpm install once (only for @deepseek-ai/schemastery, used by the Config schema), then edit and restart.

10. Capabilities can be turned off

allowScreenshots and reserveBrowserUseSlot are config options, and jev.enabled: false in a profile patch withdraws the Jev tools (browser_goal, and browser_act's intent) so decisions stay with the calling model — the settings page itself has no such switch. The Jev settings are editable under Sidebar → Plugins → `@ricardochen1996/dsh-browser-use`; allowScreenshots and reserveBrowserUseSlot are profile-patch settings.

3. Install

From npm

Requires Node.js ^22.19.0 || >=24.0.0, DSH and Chrome/Chromium. Nothing else has to be installed by hand. DSH ships a Python 3.12 runtime for its own document tools, and the plugin builds its environment with that interpreter — or with any other Python 3.12+ on the machine, or with uv when the machine has neither.

dsh plugin --profile desktop add @ricardochen1996/dsh-browser-use

Restart DSH. On its first load the plugin installs the engine it bundles into its own environment (/.venv). It creates a virtualenv with the Python it found, then pip installs vendor/requirements.txt — the sidecar's runtime dependencies, pinned by version and by the hash of every artifact in uv.lock — and finally the engine wheel the package carries, from disk. A first install downloads about 1.3 MB of wheels and takes a few seconds; when the machine has no Python 3.12+ it falls back to uv sync --frozen instead, which additionally fetches Python 3.12 (about 25 MB). The log says when it starts and when the engine is ready, browser tools wait for a running install, and browser_doctor shows how far it got. Later loads only check that the engine imports. Each plugin version builds its own environment, so an upgrade installs again. Behind a proxy, set HTTPS_PROXY in the environment DSH runs in.

Updates from the page: once installed, the bundle's page under Sidebar → Plugins grows a Check for updates control beside Uninstall. It asks the Market's public update API (dshmarket's UPDATE-API-V1) about the newest npm release, turns into Update to vX.Y.Z when one is published, follows the install's progress, and reports the outcome — a UI refresh, a DSH restart, a refusal such as a running agent, or the release-age hold the retry can force past. A Host that does not mount the Market has no such routes, and the control simply stays hidden; dsh plugin --profile desktop add @ricardochen1996/dsh-browser-use@latest remains the no-Market path.

To install ahead of time, or retry after a failure, ask for browser_doctor with install: true, or run the doctor in the installed package:

node /path/to/node_modules/@ricardochen1996/dsh-browser-use/bin/doctor.mjs --install

The install is not an npm postinstall script: DSH installs plugins with lifecycle scripts gated off, so the plugin does it when it loads. npm does not install Chrome. If installing the package into a Node project rather than a DSH profile, use npm install @ricardochen1996/dsh-browser-use; this alone does not register it with DSH. Append @0.5.1 to either command to pin a release.

If you already installed an earlier release under the old scope (@weichen96/dsh-browser-use, up to 0.5.0), or the unpublished local @rc/dsh-browser-use package, remove that plugin entry before adding the npm package; do not enable both providers. The internal id: dsh-browser-use and configuration keys remain unchanged. A profile that still sets projectPath to an engine checkout keeps using that checkout; remove it to use the bundled engine.

From a local checkout

This repo holds both the plugin (Node) and its Python half (the sidecar):

# 1. Python environment: this repo's own .venv, with the bundled engine wheel and the dev tools
uv sync

# 2. Node dependency: @deepseek-ai/schemastery, used by the Config schema (once)
pnpm install

# 3. The plugin itself: install in