dsh-agent-browser

dsh-agent-browser

English | 中文

Let the model actually use the DSH right-Sidebar browser: read a page's text, click and type in it, navigate, size the viewport, and take screenshots — from ordinary tool calls in the conversation.

It also adds a phone / tablet / desktop / custom viewport switcher, so you can see how a site renders at another size without leaving the app.

browser_navigate { url: "https://example.com" }
browser_read     { mode: "elements" }     -> refs e1, e2, …
browser_act      { action: "type", ref: "e1", text: "hello" }
browser_act      { action: "click", ref: "e2" }
browser_screenshot {}                     -> see the layout
browser_viewport { preset: "phone" }      -> re-render at 390×844

Desktop only. The tools drive an Electron <webview> guest. In Web profiles the Sidebar browser is a sandboxed cross-origin <iframe>, whose DOM cannot be read at all, so the tools answer with an explicit reason naming the Web profile instead of pretending to work. browser_tabs still lists those tabs — as mounted: true with carrier: "iframe" — so you can see that the tab exists and why it cannot be driven.

Third-party, non-invasive plugin. It mounts from your DSH profile, never touches app.asar, and leaves the built-in browser exactly as shipped.

• Install — one command, or the Plugins page
• Tools — what the model can call
• Which size to use — pick desktop or phone, not the default
• Viewport sizes — the UI side
• Compatibility — verified against DSH 0.2.1-rc.2
• Known limitations — read before filing an issue

Tools

A typical loop:

browser_navigate { url: "https://example.com" }
browser_read     { mode: "elements" }          -> refs e1, e2, …
browser_act      { action: "type", ref: "e1", text: "hello" }
browser_act      { action: "click", ref: "e2" }
browser_read     { mode: "text" }              -> confirm what changed
browser_screenshot { preset: "desktop" }       -> see the real desktop layout
browser_screenshot { preset: "phone" }         -> and the 390px-wide one
browser_close    { tabId: "tab4" }             -> tidy up when finished

tabId may be omitted whenever the Session has exactly one mounted browser tab. With several, call browser_tabs first and name the one you mean.

Reading a failed page

Three fields describe three different things, and none of them can be inferred from the others:

So mounted: true, loading: false, failed: true is a completely ordinary combination: a tab that is open and finished failing. Before failed existed, a page that never loaded looked identical to a healthy one, and the only way to tell was to read it and notice the text was empty.

unverified: true is the other half: the page could not be asked anything (its renderer is wedged, or the document is unreachable), so url and title are the last navigation's, not the current document's. Treat them as a hint, not a fact.

Both matter for screenshots, which is where the confusion cost the most: a capture is refused when either is set. A failed load paints Chrome's error page, and a wedged renderer paints a blank surface — either way the image is not the site, and returning it as a successful capture is how a page that never loaded gets mistaken for a page that "did not render". Note that failed cannot even be evaluated when the page is unscriptable, so unverified is doing real work rather than restating it.

So unverified is deliberately not just a warning label: it is the condition that makes a capture unverifiable. Inside a capture the identity probe is given more patience than elsewhere (up to 4 s) precisely so a genuinely heavy page — the several-iframes case a 15 s budget exists for — is not refused merely for being briefly busy.

Note that this is about transport failures only. An HTTP error the site itself rendered — a 404 page — is a real document and stays fully readable and screenshotable.

Waiting for the page

browser_act { action: "wait" } has two forms, and the difference matters on any page a framework renders.

A fixed sleep — { action: "wait", timeoutMs: 2500 } — sleeps and returns. Rarely what you want, for a reason worth stating: the plugin's own notion of "the page has loaded" is Electron's navigation state, which a single-page app clears as soon as its HTML and scripts arrive, long before React (or anything else) has mounted and painted. So the page reports itself loaded while the screen is still empty, and a sleep is a guess that is too short on a slow machine and wasted on a fast one. A report measured exactly that: a 2500 ms guess in a loop that ran ten times.

A named condition waits for something observable instead:

browser_act { action: "wait", selector: ".todo-list", state: "visible" }

• state: "visible" (the default) — the element is drawn. This is what "the app has rendered" usually means.
• state: "attached" — the element merely exists. Weaker, and worth knowing about: an element can exist while display: none. A report counted hidden rows as present and concluded a layout had no overflow, so the two are kept distinct.
• state: "hidden" — satisfied by absent as well as invisible, so it works for waiting out a spinner that gets removed rather than hidden.

Omit timeoutMs and the wait uses the rest of the call's own budget (still bounded, never longer than the call).

The condition is polled inside the page, so one tool call watches for it instead of a loop of calls and results. The poll uses setTimeout rather than requestAnimationFrame deliberately: rAF does not fire while a document is hidden, and a wait that silently never completes in a background tab would be a worse trap than a coarse poll.

A condition that never holds fails. It does not return ok: true with a matched: false you might skim past — the call errors, naming the selector, the state and how long it waited. Naming a condition is a way to trust the answer, so a timeout returned as success would defeat the point. An invalid selector is reported as invalid CSS immediately rather than after burning the budget, because that is a caller mistake and not the page's fault.

Which size to use

Do not screenshot at the default. The default viewport is auto, which is the Sidebar column width — typically 400–700 CSS px. That is not a device anybody uses: it fires breakpoints no real visitor sees, so a screenshot taken there misrepresents the page. Pick a size on purpose:

The tool descriptions say this too, so the model reaches for it without being told. Two ways to act on it:

• Screenshot directly at a size — browser_screenshot { preset: "phone" } applies the size and captures in one call. Prefer this: it cannot be forgotten between two calls.
• Set it once, then keep capturing — browser_viewport { preset: "desktop" } persists on that tab, and later browser_screenshot calls without a preset inherit it.

Every screenshot result reports the size it actually captured (preset, pageWidth, pageHeight, scale), so a capture taken at an unintended size is visible in the answer rather than silently misleading.

Three of those fields are easy to misread, and a report misread two of them, so they are worth stating plainly:

The misreading is worth recording because it was a coincidence: two different presets (desktop 1280×800 and custom 1440×900) both returned 1568×980, which looks exactly like a fixed 16:10 canvas. Both were 16:10 and both exceeded the 1568 cap, and the cap is the real explanation — 1568×800/1280 = 980 and 1568×900/1440 = 980. A phone capture (390×844) is well under the cap and is delivered at its own size. Likewise scale × pageWidth came out as the same 767 for both presets, which is simply the column width, not a conversion factor for the image.

When a size was requested, the result also carries confirmed. Resizing writes styles, but layout is not synchronous with that write — measuring in the same turn reports the previous size. That is exactly how preset: "phone" once came back saying pageWidth: 1280: the number was real, it was just stale. The plugin now polls until the page agrees (within a pixel) or the confirmation budget runs out:

• confirmed: true — the page was observed laying out at the requested size.
• confirmed: false — it had not reflowed; pageWidth is what it was still using. Do not judge that breakpoint from this image.

browser_viewport reports confirmed the same way, and omits it for auto, which has no target width to confirm.

Recovering a stuck or stale tab

Three things go wrong often enough to be worth naming.

A screenshot that fails while the page reads fine. This is the confusing one, and the diagnosis is now in the error. keepMounted keeps every visited tab in the DOM so the tools can reach one you are not looking at — but only the SELECTED tab of the on-screen pane is drawn, and a hidden guest is not composited. A capture aimed at a hidden tab waits for a frame that never arrives, which is why the failure "sticks", why browser_read keeps working against the same tab, and why reloading does not help (it cannot change whether a tab is being drawn).

This no longer needs the user's help. A capture aimed at a tab that is not the drawn one now borrows the selection for the duration of the call and gives it back afterwards: the app's own focus(tabId) layout operation — the one the tab strip performs on a click — moves the strip, the guest becomes composited, the frame is captured, and the tab the user was looking at is restored. Switching tabs in the Sidebar, or working in another Conversation, is therefore no longer a reason a screenshot fails.

What remains is the one case that genuinely cannot be borrowed: a Conversation whose Sidebar is not on screen at all. The layout operations act on the mounted Session, so there is no surface to focus, and the call says so instead of waiting out a transport timeout. The result is a fast, actionable error rather than the old 60-second silence:

its Conversation is not on screen, so its Sidebar cannot be focused. Open this Conversation, then retry.

Two properties keep that borrowing safe. It happens only when the DOM confirms the tab is not drawn — never on a shell whose attributes this code cannot read, so ordinary captures gain no new failure mode. And the restore runs in a finally, so a capture that throws still puts the strip back where the user left it; a failed call never leaves the Sidebar on the agent's tab.

Every failed capture still says which case it was, from the host DOM: an ancestor is hidden (the Conversation is not on screen), the Sidebar is collapsed, another tab is the one being shown (named), or visibility is ruled out because the tab really is the drawn one. When the shell exposes none of those signals the diagnosis says nothing rather than guessing — "ruled out" is only claimed from evidence.

A page that has stopped answering. Every guest call is bounded by the plugin's own budget (6 s for a read or an action, 15 s for a screenshot — a capture forces a whole repaint and a heavy page legitimately needs longer). A stalled renderer produces an error saying so instead of a 60-second stall. Note what the budget does and does not buy you: it bounds how long the browser waits before giving up. The Inspect channel then has its own timeout, and when the Client reports a failure the registry keeps that as a diagnostic and does not settle the query — so the caller still waits out clientInspectTimeoutMs before being told. Both numbers appear in the message and neither is the cause; the plugin's own reason is the one to act on.

A change that appears not to have taken effect. A dev server that sends no cache headers — python -m http.server is the usual one — lets Chrome serve the old CSS and JS under heuristic caching, so fresh files look like they never applied. Numbers that "did not change" are the tell:

browser_act { action: "reload", ignoreCache: true }

The result reports ignoreCache, which is false if the carrier had no cache-bypassing reload, so a plain reload is never passed off as a bypass.

Tabs opened while investigating something used to accumulate with no way to remove them. They can now be closed:

browser_close { tabId: "tab4" }

Closure is confirmed against the tab list before success is reported, so a close the Sidebar ignored is an error rather than a false confirmation.

Viewport sizes (phone / desktop / custom)

A browser tab normally renders at the Sidebar column's width. Switching the viewport makes the page lay out at a real phone, tablet or desktop width instead, so a site's own media queries fire and you see the layout that size really produces. See Which size to use above for what to pick — the short version is that auto is the column and you almost never want it for a screenshot.

Three ways to drive it:

• Right-click the tab — the tab strip has no "⋯" button. The kit opens a tab's menu on right-click (its onContextMenu handler), and that menu is where these items appear, under Viewport size. The current choice is ticked. With only one tab open the menu is otherwise just Close tab; these items are what make it worth opening.
• Settings → General → Browser viewport — the default that new tabs inherit.
• Shortcut — browser.viewport.cycle cycles fit → phone → tablet → desktop for the focused browser tab. It ships unbound so it never steals a chord; bind it yourself in Settings → Shortcuts and search for "viewport".

browser_viewport exposes the same control to the model. Reading it reports both the preset and the CSS pixel width the page is actually laying out at:

{ "tabId": "tab3", "preset": "phone", "pageWidth": 390, "pageHeight": 844,
  "targetWidth": 390, "scale": 0.62, "mounted": true }

The read resolves an implicit tab the same way the write does, so tabId is always reported — omitting it targets the Session's only browser tab, and naming a tab that does not exist is refused with the same error the other tools give.

How it works, and why not zoom

The guest is an Electron <webview>, and that element's box is the page's viewport: a 390px-wide element makes the document lay out at 390 CSS pixels. To show a viewport wider than the column, the element is scaled down for display only, inside a clipping container, and the scale is recomputed by a ResizeObserver when the Sidebar is resized. The page's layout stays truthful; only the presentation shrinks. pageWidth in the read-back is the guest's own window.innerWidth, so it always reports what the page believes.

<webview>.setZoomFactor() is deliberately not used. Chromium's zoom policy is same-origin, so zooming one tab would also zoom any other tab showing the same site — tabs in one workspace share a storage partition. Element box plus a CSS transform keeps every tab independent.

Placement

A sized page is centred in the column, both across and (when it fits) down, so it reads as a device preview rather than a page jammed into the top-left corner. Centring is done with translate(...) placed before scale(...) in the transform list: the offsets are therefore in unscaled column pixels while the scale still happens about the element's top-left corner.

Two deliberate limits:

• Flexbox is not used for centring. Flex would centre the element's layout box, which is the full unscaled target (1280px inside a ~650px column), so it would push the scaled result off to one side while the translate did the same job again. One mechanism only; leftover flex centring is cleared on every apply so an already-styled container is corrected in place.
• A page taller than the column is top-anchored, not centred. Vertically centring an 844px-tall phone page in a shorter column would clip both ends and hide where the document starts.

Consequences worth knowing:

• At a desktop width in a narrow column, text is genuinely smaller — that is the honest cost of seeing the real desktop layout. Use a custom size (240–3840 × 320–2160) to trade layout for legibility.
• A tab that is open but not mounted cannot be resized yet; the choice is stored and applied the moment you click that tab. A restored Session re-applies saved presets shortly after load.
• Sizes are remembered per tab in localStorage under dsh.agent-browser.viewport.v1 and survive a restart.
• The height is the preset's real height too. A "desktop 1280×800" tab lays out at exactly 1280×800 and is then drawn scaled-down inside the column. Scaling the height up to fill the column was tried and rejected: it stretched the viewport (that preset laid out 1571px tall) and silently broke 100vh and every height-driven layout. The scaled result is therefore often shorter than the column — which now shows as balanced space above and below the page.

Configuration

In the profile's cordis.patch.yml:

- id: dsh-agent-browser
  name: dsh-agent-browser
  config:
    # Allow browser_eval to run script in the page. The TOOL is always
    # registered; with this false it refuses and explains how to enable it.
    enableEval: false
    # Character budget for browser_read text/html.
    maxReadChars: 20000
    # browser_screenshot downscales to at most this width before delivery.
    screenshotMaxWidth: 1568

browser_eval is deliberately visible either way. Registering it only when enabled was the earlier design, on the reasoning that a deployment should not advertise a capability it has not accepted. Two independent agent reports showed what that costs: needing layout geometry (getBoundingClientRect, scrollHeight, getComputedStyle) — exactly what eval is for — neither agent could see the capability existed, and each rebuilt the same workaround instead, writing same-origin iframe probe pages into the project under test. That polluted a repository whose .gitignore did not cover them and travelled into the build output; one round shipped three bugs of its own, including a scrollbar artifact that nearly produced a false layout conclusion. An invisible capability is not a safe one, it is an unguessable one — so the tool is registered, refuses when the switch is off, and names the switch. Refusing grants nothing: the guard that closes the cordis_inspect_query route to the provider's eval method is unchanged, so the switch remains a real boundary rather than a hint.

The Inspect channel also has a process-wide client-query timeout, defaulting to 10 s. A slow page can exceed it, so raise it on the runner that owns the registry:

- id: cordis-host-runner
  name: '@deepseek-ai/dsh-cordis-host-runner'
  config:
    clientInspectTimeoutMs: 60000

Each browser call spends its own budget waiting on the page — 6 s for a read or an action, 15 s for a screenshot — deliberately under that default so a slow page answers "did not answer within 6s" instead of turning into an opaque transport timeout. Both cover the whole call, not one step of it: browser_act { action: "wait" }, navigation settles and the pause after a click all draw from the same deadline, so no single call can outlast the caller.

Be precise about what that budget buys, because a report read it as the total wait: it bounds how long the browser waits before giving up. It does not bound the call. When the Client reports a failure, the registry records it as a diagnostic and keeps waiting for a success, so the caller still waits out clientInspectTimeoutMs. That is why a failure can show both timed out after 60000ms and did not answer within 13s — the first is the channel, the second is this plugin, and only the second is a diagnosis. Raise clientInspectTimeoutMs (above) if you want a longer ceiling to spend; the plugin's budgets stay as they are, since they are not configurable today.

How it works

model → browser_* tool (Host)
          └─ ctx.cordisInspect.query('client', 'AgentBrowser', <method>, …)
                └─ WebSocket/RPC → the page → Client Inspect provider
                      └─ finds the tab's guest by carrier
                         [data-sidebar