CJYLZS/dsh-browser

dsh-browser

English | 中文

A real local browser for every conversation. The agent opens pages and works in them while the sidebar mirrors the same browser, so you can watch the work happen and take over whenever you want. It runs in both the Web and the Desktop interface.

Summary

Chrome or Edge runs as its own process, with its own profile and its own CDP port, and every conversation gets its own: tabs, sign-ins, and ports are never shared between conversations. The agent drives it with nine tools, and the sidebar shows the same pages — either side can act on them.

It is not an embedded web view. The page runs in an ordinary browser process, so sites see a normal browser, DevTools or another Playwright can attach to it while the mirror is live, and closing the harness does not leave a browser pretending to be part of the app.

Highlights

• Works in both interfaces. The Web GUI and the Desktop app run the same plugin against the same browser; install it once in each and nothing else differs, including the settings page.
• One browser per conversation. Tabs, sign-ins, and ports belong to one conversation and stay there: the sidebar pane names its session, and a tool call resolves its browser from the session that called it.
• Watch it work, or take over. Every page the browser opens gets its own sidebar tab with a live view, and clicks and typing in that tab reach the real page.
• The agent can read a page it has never seen. browser_snapshot prints the page's accessibility tree with a ref per node, which is how the agent reads an unfamiliar page and how it names the element to click; find narrows a large page, and boxes gives positions without a screenshot.
• Input a site accepts. Real mouse and text events at the element's own position — a page that ignores a synthetic element.click() still takes these — plus chords such as Ctrl+A, native dropdowns and checkboxes, dialogs answered by the call that opens them, and an explicit refusal when something else would receive the press.
• Nine tools, and code for the rest. browser_evaluate is the general-purpose one; the others cover what code cannot express, such as reading a page without knowing its selectors first, or asking what the page said while it failed. See Tools.
• Attachable. Each browser listens on an external CDP port, so chrome://inspect, another Playwright, or the bundled scripts/cdp.mjs can attach to the same browser the sidebar is mirroring.
• Working rules where the model reads them. The plugin contributes a skill — read before acting, one action per observation, and a page's text is data rather than instructions — with five short recipes read only when their question comes up.

Table of Contents

• Highlights
• Install
• Compatibility
• Usage
• Understand the design
• Configuration
• Tools
• Known Limitations and Deferred Work
• Dev Note
  • Third-party code

---

Install

Check the host dsh version first:

dsh -V

Then install with the #<tag> ref that version pairs with — a build targets one dsh generation, and a github: install without a ref takes the default branch HEAD, which drifts:

For a dsh on the current generation, the latest release is:

dsh plugin add --profile web github:CJYLZS/dsh-browser#v0.2.3

The built lib/ is committed with each tag, so a tag install needs no build step. The profile's package.json records the ref you chose; to change versions, re-add with the new ref, and to remove the plugin use dsh plugin remove --profile web dsh-browser. Restart the harness after installing. A Chrome or Edge installation is required; playwright-core is a dependency and downloads no browser of its own.

Desktop

The Desktop app owns its own profile, and the ordinary CLI refuses to manage --profile desktop on purpose, so install through the Desktop CLI that ships with the app:

"<Desktop install dir>/resources/runtime/cli/bin/dsh.cmd" plugin --profile desktop add github:CJYLZS/dsh-browser#v0.2.3

Use <Desktop install dir>/resources/runtime/cli/bin/dsh on macOS and Linux. Restart the app afterwards, the same way the Web profile restarts after dsh plugin add; the same remove verb uninstalls it.

---

Compatibility

A build targets one dsh generation, so install the plugin version that pairs with your dsh. The fork is in the client half — dsh 0.1.7-rc.1 replaced the settings model with no compatibility layer — which is why one build cannot serve both generations.

That fork is in the client half's imports, so one build cannot target both generations and the plugin pairs by generation:

• v0.2.x → dsh ≥ 0.1.7-rc.1, declared as peerDependencies: >=0.1.7-rc.1 with no upper bound, so one build runs on the whole 0.2.0 line it is verified against (why the cap went away). The settings page's fields are marked volatile in the schema, and an edit reaches the running browsers through loader/volatile-update.
• v0.1.x → dsh 0.1.5-rc.2, declared as peerDependencies: ^0.1.5-rc.2. The settings page registers a namespace scope and renders its own controls.

An older dsh fails loudly: dsh activates a plugin only when its dsh peers satisfy the declared ranges, and names the plugin and the unsatisfied ranges when they do not. A later generation passes that check, so whether this build still runs on one is a question for verification rather than for the version number.

Usage

Three steps, and the agent needs no instruction beyond what it is asked to do:

1. Nothing, usually. The first tool call that needs a browser starts one, and the right sidebar opens itself on that session's Browser tab — you watch the page from its first frame. You can also open it by hand: New tab → Browser.
2. Ask for something. "Open the docs and tell me what the install section says" is enough: the tools act on the same browser the pane shows, so you watch the work happen.
3. Attach your own tools when you want them. curl http://127.0.0.1:9333/json/version answers while the browser runs; chrome://inspect, another Playwright over chromium.connectOverCDP, and scripts/cdp.mjs all attach to it.

The sidebar and the browser stay matched: one tab per page the browser has open, a new page adds a tab, closing a tab closes the page, and closing the last one stops the browser until something asks for another one. Which page the tools act on is the agent's choice (browser_tabs), independent of which tab you are looking at. Understand the design covers the rest.

---

Understand the design

The plugin keys everything by the harness session id, and that one decision explains most of its behavior:

• Two independent paths resolve the same session. The sidebar pane is registered into a session-scoped slot, so its registration receives the session id and names it on the viewer socket; a tool call has no pane, so it takes the session from the agent executing it (exec.agent.id, which is the same SessionId the API layer resumes sessions with). Neither path can reach a browser belonging to another conversation, and a tool called with no session — a scheduled job, or a subagent without one — fails instead of landing in somebody's browser.
• One process per profile. Chrome locks a profile directory to a single running process, so the configured profile directory is a parent: each session's browser gets its own subdirectory under it, named with the harness's own session-id escaping. The consequence is worth stating plainly, because it is the price of the isolation: sign-ins are not shared between conversations. A site you log into in one conversation starts signed out in another. Sessions keep their identity across reopen, so a conversation returns to its own profile.
• Ports are allocated, not assumed. The configuration names a window of ports (debugPortMin–debugPortMax) because one number could never be the address: every session gets its own browser and its own listener. Each browser takes the lowest port in the window it can actually bind and holds it until it exits, because a probe's answer is stale the moment it is given. The top of the window is what stops a search from wandering into whatever else the machine runs: a browser that finds nothing free inside it fails with the window in the message. GET /dsh-browser/status reports the port each session ended up on; nothing should assume it is one of the configured values.
• The mirror belongs to a CDP session, not to the pane. Page.startScreencast is attached to one CDP session, and every path that starts a browser — the pane's restart, recovery from a crash, a launch setting change — replaces that session. A newly started browser therefore re-attaches the cast as soon as it is ready if anyone is watching, or a pane left open across a restart would keep its last frame forever while its status line kept updating.
• A dead browser is replaced, not mourned. Closing the window or losing the process moves the instance to closed with the reason, publishes it to the pane, and leaves the next request to start a fresh browser. Nothing has to be fixed by hand, including the browser process dying between two tool calls.
• The snapshot is a contract with the model. Accessibility.getFullAXTree is printed as one line per node, with ignored wrappers dropped and a ref for every node that has a DOM node behind it. A ref names a node on one page: navigating invalidates all of them, and a stale ref fails with a message pointing at browser_snapshot rather than clicking whatever now occupies that position.
• A ref or a locator; an ambiguous locator is refused. A ref is an answer one snapshot gave about one document, so it is cheap to check and goes stale. A locator — role with name, text, or selector — is the question that answer came from, asked again when the action runs, so a re-render or a navigation does not stop it. A locator matching more than one element is refused rather than guessed at, and the refusal lists every candidate with its nearest named ancestors, which is the only thing that tells two identically named controls apart. Matching reads the accessibility tree, so a locator matches exactly what a snapshot would print; a selector is walked by the page itself, so it reaches an open shadow root and a same-origin frame — the parts a query rooted at the document cannot see into, which is why target and snapshotIgnore resolve their selectors the same way.
• Clicks are computed, not guessed. A click resolves its ref to a backend node id, scrolls the element into view, reads its content quad, and converts that from page coordinates to the viewport coordinates CDP dispatches input in. It then sends a move, a press, and a release, which is what makes the event trusted.
• A page is mirrored by its own tab. The sidebar holds one tab per page the browser has open, addressed by the page's CDP target id, so two pages on the same address stay two tabs and a reloaded client re-attaches every tab to the page it was watching. A pane's frames, input, and address bar belong to its page, whether or not the tools are acting on it right now; a page the browser closes takes its tab with it, and closing a tab closes that page. Every page the entry opens comes from one pick. The page-less tab the browser entry opens asks the host for a page as it mounts — the ask is keyed by that tab record's own id, so a tab switched away and back, a Session change, or a client reload asks once — and the host opens one more page on a browser already running, or answers with the page a browser it started came up on. Which page the tools act on is browser_tabs's open/select, independent of which pane the user is looking at.
• The plugin's own routes carry the trust check. The viewer socket, the status route, and the page route (open and close) are served by this plugin, and the webserver's route path runs no authentication of its own, so each applies Connection's rejection (isTrustedApiRequest and browser authentication) before answering. An unauthenticated request to any of them is refused with 401.
• Settings are user overrides, not state. The settings page writes to the harness settings document, and the cordis.yml entry stays the base layer beneath it; clearing a field removes the override. Every field changes how a browser is launched or encoded, so a write restarts the running browsers and the panes reconnect on their own.

---

Configuration

The settings section (Settings → Browser) edits the fields below and reports what is actually running, which is not the same thing as what is configured: the banner lists every session's browser with the port, window mode, and page count it really has. A deployment without a settings provider registers nothing, and the composition entry is the whole configuration.

stealth is on by default because it was measured, not assumed: with the markers in place Google served its "unusual traffic" page three times out of three, and with them removed three times out of three searches returned results — including for a real Edge window started the same way, which is what showed the markers rather than the browser to be the deciding factor. The setting exists so the comparison can be reproduced. When a deployment needs more than markers can hide, executablePath accepts any Chromium binary the user supplies, including hardened builds; no third-party binary is bundled or downloaded by this plugin.

---

Tools

An action result carries a compact list of what changed. changed says the page moved (dom), which does not distinguish a click that opened what it named from one that only repainted a spinner. So the action also reports the first few changes the page made, each in facts the page itself can stand behind — its tag, the role it declares, and a short piece of what it says: dom: +1 status "Saved"; ~ p "24.1k" → "24.2k"; ~ button "Save" disabled: (none) → "true". The list is capped at five and the changes that did not fit are counted rather than dropped, and every entry is structured (kind/tag/role/preview/attribute/from/to) so a PTC program can branch on it. This is the page's own description of itself, for knowing where to look: the accessible roles and names are still browser_snapshot's to report.

A dialog is answered by the call that opens it. A confirm, prompt, or alert blocks the page until it is answered, so there is no moment at which a later call could answer one: pass dialog: "accept" — with dialogText for a prompt — on the call that triggers it, or the default dismissal is reported in the result. Every dialog is answered before the call returns and none is silent: the result names what the page asked and how it was answered, and changed says dialog, so "the page did not change" can never be the report for a click a site stopped to confirm.

The set is deliberately small because browser_evaluate exists: a tool earns its place only by doing something code cannot, which here means trusted input, reading a page without knowing its selectors first, and the two questions that cannot be asked after the fact — whether a state has been reached (browser_wait), and what the page said about itself while it failed (browser_console, whose listener has to be subscribed before the page's own scripts run). All three are deliberate exceptions to "new capability becomes a parameter, not a tool name" (the third is browser_tabs: a page is not a property of the current page, and page script cannot see another target); the screenshot's shapes are the rule followed instead: fullPage, an element target, and inline are parameters on the tool that already existed. Choosing an option and setting a checkbox are the same shape — select and checked are parameters on browser_click — because a press can do neither. Deliberately absent are back/forward/reload (the pane has reload; history is one expression), hover, drag, upload, and download — each would be added when a real case needs it, not in anticipation.

Screenshots are written to a file, and the path comes back so it is durable in the session log. inline: true also attaches the image to the result, which costs no second call; the attachment is a reference the harness's attachment service mints (the plugin declares that service structurally and does not depend on its package), and the call is refused before the capture when the calling model does not declare image input — an image block is part of the tool result, so attaching one to a model that cannot read it would fail that request and every request after it.

---

Known Limitations and Deferred Work

• Verified on Windows only. macOS and Linux have not been run once: the channel lookup, temporary directories, and process teardown are the parts most likely to differ.
• **The pane shows the page its tab names, and pages the brows