dsh-browser-crossplatform

中文版见 README.zh.md

dsh-browser-crossplatform

An MV3 extension + bridge plugin that lets the dsh desktop app read and operate a browser tab: pages are handed to the model as text (numbered actionable controls), and when image viewing is needed there is a separate optional channel, off by default. This is a derivative rewrite of Lum1104/dsh-browser, restructured around two priorities — high compatibility and trimming bloat — while balancing stability with efficiency.

You can find this bridge plugin in the plugin marketplace under dsh-plugin, the marketplace's conventional topic tag.

What sets it apart from similar projects: cross-platform (Windows / macOS / Linux, zero platform branches in the source) · the root package has zero dsh dependencies (the bridge declares all of dsh as peerDependencies) · image viewing is optional and off by default · one unified version number across the whole repository · and benchmarks and CI that compare the extension against a local Playwright baseline in pairs.

Installation

Three routes — pick any one:

# From the npm registry (published: dsh-browser-crossplatform@0.38.7):
dsh plugin --profile desktop add dsh-browser-crossplatform   # the CLI dsh web uses --profile web

# Or straight from this repository, with no registry involved:
git clone https://github.com/youbaiyun/dsh-browser-crossplatform
dsh plugin --profile desktop add link:<clone>/packages/bridge

Step-by-step foolproof instructions (including "how to confirm it is installed") are in docs/INSTALL.md.

The extension itself is installed from the browser extension store (or load the unpacked build yourself following the build section below); the package above is the bridge plugin, and its settings page lives inside dsh, titled 「dsh 浏览器设置」.

Structure (five packages share the same version number 0.38.7)

packages/protocol   zero-dependency wire protocol (frame validation, capability/authority split)
packages/bridge     dsh bridge plugin (WebSocket server + browser_* tools, runs inside dsh)
extension           the Chrome/Firefox MV3 extension itself
skills/             the troubleshooting skill your AI agent reads when the panel misbehaves:
                    one symptom → one section → a repair, without the user opening devtools

Compatibility range (all targets and minimum versions)

The bridge has only ws and @deepseek-ai/schemastery as runtime dependencies, and both are platform-independent; the extension bundles its two panel-only dependencies (marked, dompurify) into the build. CI builds both browser targets and runs the full test suite (including end-to-end cases that actually launch Chromium); Windows has been verified locally; macOS is not covered by CI, but it is POSIX like Linux and the platform-specific surface is that one module.

The loopback shortcut is bound to named extension ids. The bridge normally authenticates with a bearer token, but a loopback upgrade from the extension itself skips it so discovery stays zero-config. That exemption is not "any chrome-extension:// origin" — every other extension on the machine has one of those too — but an exact match against extensionId, a comma-separated list whose default names two ids: kdhkdgfcinfkmogifamoapmheihhcjfk, which this repository's manifest key produces, and agpipnjijkpomaannkijkilggoffdiaf, which the Chrome Web Store assigned. Both are needed because the store refuses a manifest carrying key, so a development load and a store install present different origins. Add your own id to the list for a build of your own, or set extensionId: '' in the plugin config to require the token on every connection, including loopback.

Why phones and tablets are not supported: Chrome / Edge for Android does not support third-party extensions; Firefox for Android has no side panel UI; and the bridge is loopback-only where it matters — the token-free path and the privileged gateway methods (host.pickDirectory, host.openPath, settings.*, credentials.*) are both gated on a loopback remote (packages/bridge/src/server.ts), and a phone would be connecting to 127.0.0.1 on a different machine. A non-loopback remote with a valid token can still use the ordinary session methods that dsh web --host exists to serve; what it cannot do is reach the privileged ones.

Key differences from the original

Core mechanisms

• Text snapshot + action execution: no screenshots, no recognition step (at the protocol layer, textOnly: true). Pages are rendered into structured text: title/URL/body (readability-lite) + a numbered interactive inventory (including ARIA role controls) + form fields (including masked/checked/required), with support for delta diffs and region partial snapshots.
• Safety invariants: the values of sensitive fields (type=password, autocomplete=credit-card|cc-*, id/name/aria-label matching password|passwd|credit|card|cvv|cvc|secret|pwd) are always masked as ••••; the accessible name never uses the input's current value (only the value of submit/button/reset inputs counts as a name), and unit tests enforce this.
• Tool surface (17): browser_snapshot / click / type / press / scroll / navigate / open_tab / list_tabs / follow_tab / close_tab / back / forward / reload / get_text / wait / launch / describe_image. Full parameter list: delta/region/replace/amount/selector/ms/active/tabId/index/text/key/direction/url, plus frame on the 7 frame-scoped tools. browser_describe_image answers only while recognition is on; browser_launch is the one tool that runs on the desktop side, because it is the only one that can work with no browser running.
• Starting a closed browser (browser_launch, and before every tool): tool execution lives in the extension, so with the browser closed there is nothing to dispatch to — the tools used to fail with a bare bridge-closed. Now a call with no connection first tries to start the browser and then reports what actually happened. It launches the browser you already use: the executable comes from the platform's own record of which app opens a link (the Windows UserChoice registry entry, or macOS LaunchServices), falling back to detection, and there are no extra flags — no --user-data-dir (that would be a second browser with none of your tabs or logins) and no --load-extension. If that browser is already running with a window, nothing is launched at all (a second start would only open a window in the existing process while discarding the flags), and the answer says which browser to enable the extension in. browserUserDataDir + extensionPath exist for the development case, a named profile where an unpacked build is genuinely loaded.
• What it cannot do: install the extension. A command-line load lasts one session and installs nothing — closing the browser loses it — and since Chrome 137 branded Chrome/Edge builds ignore --load-extension altogether (Chromium and Chrome-for-Testing still honour it). So the extension has to be installed in the profile once (store, or "load unpacked" on the extensions page), and after that a normal start reconnects by itself. The bridge checks <profile>/Extensions/<id> in Chrome/Edge/Brave/Chromium profiles and says plainly which of the two situations the user is in rather than offering a step that cannot help.
• Frame routing: a snapshot combines the main frame with all accessible iframes, and subframes are labeled [frame N] <origin>; the background remembers N → frameId and routes later tools carrying frame to the same frame; each frame renders its section within the same negotiated budget, and the body is truncated first (mainBudget = maxChars × 0.5) so a long page cannot swallow the inventory.
• Capabilities added beyond upstream (additions, not removals): browser_type can fill a <select> (matching in order by option value → visible label → 1-based index, listing the available options on failure) and set a checkbox/radio with true/false; browser_wait supports wait conditions (selector / text, failing with the timeout error code if they never appear) instead of only a fixed delay.
• Automatic snapshot after navigation: the content script announces when the new document is ready (the equivalent of the original's DSH_CONTENT_READY), and after navigation-class actions (navigate/back/forward/reload) the background performs the handshake, waits for the new document and snapshots automatically, pushing the result to the panel — the model does not need another round trip.
• Image manifest (useful even with image viewing off): the ported pipeline originally discarded bare <img> elements entirely — images are not in the interactive selectors, and innerText carries no alt either. Now an Images: section is added at the end of the snapshot: in document order, grouped by the nearest heading, each line rendered as [N] kind WxH "label" near="…" text-described [unavailable] → src ⟦iN img:state⟧ (content/snapshot.ts, renderImage). N is the same interactive index the inventory uses, so the number in the section is the number to browser_click (for the images that are clickable at all); text-described marks an image whose author already supplied text, and the trailing marker carries the recognition state so the section and the inline marker always agree. Positional information comes from the DOM, so the model never needs coordinates. The marker is nonce-escaped, so a page cannot forge the section or the state it carries. Image types covered: <img> (including currentSrc, so a srcset candidate is reported rather than the fallback), <video poster> (the cover is an ordinary image URL and goes through the same fetch path), inline <svg> (no bytes to fetch, so it is reported as kind=svg and never handed to the fetch path), and CSS background images — inline background-image declarations are collected exactly, while class-declared ones go through a bounded computed-style scan: every element in the region is visited until BACKGROUND_SCAN_LIMIT (1500) elements or MAX_BACKGROUND_IMAGES (20) background images, whichever comes first, with elements already collected as img/svg/video and script/style tags skipped. <canvas> is not collected: a canvas has no address to fetch, and reading it back would mean reading page memory, so it is left out rather than listed as unreadable. Rendering happens against the background's answers: the content script collects the images and hands back ImageView[] out of band, and it renders the Images: section from those views while the background holds the recognition results and the cache — because recognition is asynchronous, the background is the side that knows the answer and fills each marker's state afterwards. Each iframe renders its own section, so the indices are frame-local. The budget is the item budget, not a second one: images are capped at a third of maxItems (floor 4) and share the interactive inventory's numbering and cap, so a sixty-thumbnail page cannot crowd out the controls (content/snapshot.ts, imageCap).
• Image fetching and recognition cache (background): background/image-fetch.ts fetches images inside the service worker — the extension context holds host permissions, so it can read cross-origin image bytes directly that a content script could never read (a content script canvas would be tainted); it passes credentials:'include' to cover media behind a login. It does not decode, resize or re-encode: it returns the bytes as they are, or says why it could not, and a body over MAX_IMAGE_BYTES (4 MB, since base64 inflates by a third and the payload crosses the WebSocket) is not carried inline — the caller falls back to handing the URL to the desktop. Failures are classified rather than thrown (too-large / unsupported-url / fetch-failed / empty, with a permanent flag for the ones that cannot succeed later in the session), because the model must be told "there is an image here, but it cannot be read" rather than seeing nothing at all. background/image-cache.ts caches results and failures by image identity (URL), persisted to chrome.storage.local for up to 24 hours with the 200 most recent descriptions kept and transient failures retried at most twice — the cache exists for consistency first (the same logo in thirty rows must not yield thirty different sentences) and for cost second, and it is not memory-only because the MV3 worker is stopped and restarted freely. background/vision.ts is where the recognizer plugs in: direct-to-cloud and via-desktop-bridge differ only in that one injected function, while fetching/caching/rendering are exactly the same.
• Two recognition transports, desktop relay preferred: the protocol adds an image.call / image.result frame pair, and hello.ok's policy gains imageRecognition. The extension fetches the image itself first (only it carries the login state) and sends the bytes down with the frame; when it cannot, it hands the URL to the desktop, which fetches with a network stack unconstrained by CSP/host permission/enterprise policy — that is the real value of desktop relay, and why the frame carries a source field instead of one fixed type. On the desktop side vision.ts calls the configured chat-completions model (deepseek-flash by default, images as data URLs, thinking off, usage read back to confirm the setting took effect), and image-relay.ts fetches the bytes or falls back to the URL and classifies failures. Credentials stay on the desktop: hello.ok reports imageRecognition: false when no vision client could be built, which happens when neither visionApiKey nor the desktop's own DEEPSEEK_API_KEY credential resolves. background/vision.ts picks the transport per request: the desktop relay wins whenever the desktop advertised it, and the extension's own direct path is used otherwise.
• Direct external API connection (the recognition step is invisible in the UI): background/vision.ts (DirectRecognizer) lets the extension call the external API itself, reading its configuration from chrome.storage.local (visionEndpoint / visionApiKey / visionModel; the timeout and the thinking switch exist only on the desktop-side plugin config), and adds no UI at all — the panel shows the image-viewing tier and nothing else about recognition. A real failure (429 etc.) is final rather than falling through to the other transport, so you never pay twice for the same image. The vision-only settings live in src/settings.ts next to the rest, and the panel never exposes them. Key guarantee: this recognition step creates no session, writes no file and produces no conversation entry — it is just an ordinary outbound call in the background; the description memo goes to chrome.storage.local (24-hour TTL). Both paths share the same prompt and parser in protocol/src/vision-contract.ts, so which side calls the model changes neither the question asked nor what counts as a valid answer.
• Panel markdown: model replies are rendered with marked and sanitized with DOMPurify; during streaming the reply stays plain text (avoiding re-parsing on every frame) and is rendered once it is complete; links in replies open in a new tab without leaving the panel.
• Approval confirmation: state-changing actions can ask the user first (the unrestrictedBrowserAccess switch, trusted origins, allow-once). The switch ships on, so a fresh install is not asked; turning it off restores the prompts. While it is on it also overrides the page-sharing choice, so "ask before reading the page" does not apply.
• On-demand injection: no manifest content_scripts is required for the controlled tab — scripts are injected with chrome.scripting into the controlled tab and re-injected after navigation; the manifest's declaration exists so that a tab open before the extension was installed or reloaded is already covered.
• Tab affinity: tools are bound to a single controlled tab. What a manual tab switch does is the follow-tab setting: follow (the shipped default) moves the binding to the tab you went to, ask blocks and asks whether to keep the current tab or follow, keep stays put.
• Session bridging: panel prompts are sent to the desktop dsh via rpc frames, and assistant text streams back.
• Bridge plugin: a token-authenticated WebSocket on /ext/bridge, a hello handshake negotiating caps/policy, browser_* tools dispatched to the extension as tool.call frames, and privileged gateway methods always rejected for non-loopback remotes.

Image viewing (recognition)

There is only one switch in the UI: panel → settings → image viewing (off / low / standard / enhanced). It is off by default; once it is on, the image you ask about does leave this machine for whichever model is configured (see below).

Where the calls go is deployment configuration, not a setting — the panel shows the image-viewing tier and nothing else about recognition, so the endpoint, model and key are set by whoever deploys it (the desktop plugin config, or extension storage). The panel's tests hold that line: extension/tests/control-page.spec.ts asserts it exposes no control the browser cannot honour, and the tier select is the only recognition-related field it renders.

Desktop relay (recommended): in the desktop profile's cordis.patch.yml, add a config key to the bridge plugin; the address and model already have defaults (https://api.deepseek.com/v1 / deepseek-flash), and thinking defaults to off:

- id: bridge-browser
  disabled: false
  config:
    visionApiKey: <key>
    # Optional; the id must be the API id, not the display name —
    # `deepseek-flash`, never `DeepSeek-V4.1-Flash` (the API rejects that with 400).
    visionModel: deepseek-flash

Browser direct: there is no UI entry point; just write the settings into extension storage for your deployment (the manifest's connect-src already allows https: and http: for any host, so no manifest edit is needed):

chrome.storage.local.set({ dshSettings: { visionEndpoint: 'https://…/v1', visionModel: '…', visionApiKey: '…' } })

tools/vision-stub.mjs and tools/vision-proxy.mjs are development tools: the former measures local overhead, the latter lets the extension go to the cloud without changing the manifest. Normal use does not need them.

Commands

pnpm install
pnpm -r run typecheck   # typecheck the whole repo (protocol + bridge + extension)
pnpm -r run test        # all unit tests (the bridge's e2e needs a Chromium; see below)
pnpm --filter dsh-browser-extension run build           # Chrome → extension/dist
pnpm --filter dsh-browser-extension run build:firefox   # Firefox → extension/dist-firefox
pnpm --filter dsh-browser-extension run bui