dsh-scrapling

dsh-scrapling

A web-scraping extension for dsh: Scrapling as agent tools — JavaScript rendering, CSS/XPath extraction with self-healing selectors, sessions with login state, a background crawler and page Q&A. You supply the interpreter and the browser engine; everything runs in one resident Python process.

Contents

• What it does
• Install
• Verify it is live
• What it does not provide
• Enabling it in dsh
• The tools
• Replacing the built-in fetch and search
• Settings
• Proxies
• Security
• Data and privacy
• Known boundaries
• FAQ
• Development
• License

What it does

dsh ships web_fetch and web_search. The built-in web_fetch is deliberately honest and minimal: a plain HTTP(S) GET with no JavaScript execution and an honest User-Agent. That is a good default — and it means a single-page application yields only its loading shell, and there is no way to ask for "just the product table". Measured on a page whose content exists only after JavaScript runs:

Pipeline · Text obtained raw HTML to text (no JS) · SPA Shell / LOADING_PLACEHOLDER Fetcher (curl_cffi, still no JS) · SPA Shell / LOADING_PLACEHOLDER browser render (patchright) · full rendered DOM, Markdown table, structured rows

This package adds what the built-ins structurally cannot do:

• Browser rendering — JavaScript execution via patchright.
• Structured extraction — CSS/XPath selection returning data, not a wall of Markdown.
• Self-healing selectors — Scrapling's adaptive mode relocates an element after the site's markup changes; a selector that returns None after a class rename returns the relocated element with adaptive on.

Everything runs against one long-lived Python process. That is not an optimisation: Scrapling's session classes keep cookies, the browser engines keep a page pool, and adaptive keeps a SQLite fingerprint database — a process per call would throw all of that away. Tools are registered conditionally: the plugin probes what the environment can actually do at startup and registers only what will work, so the model never burns a turn on a tool that was guaranteed to fail. Every tool renders its result row through the official tool-call view slot. scrapling_answer is the only tool that spends another model call — see The tools.

Install

dsh plugin --profile web add @jayyuen66/dsh-scrapling

Remove with dsh plugin --profile web remove @jayyuen66/dsh-scrapling. The package is on the public npm registry, so installation needs no credentials and no scoped-registry line.

• Requires Node.js `>= 22.12`. This is an ESM-only package; that Node floor is what makes require() of an ESM entry work at all, which is why engines.node is declared.
• Requires host `dsh` at the `0.2.1-rc.3` line. The requirement the host actually reads is engines.dsh (^0.2.1-rc.3), and it decides alone; a host on a later line is refused and needs the explicit risk grant dsh plugin --profile allow-version @ --dsh-version --accept-risk.
• Requires Python 3.14+ with Scrapling installed — in the repository's _env/.venv, or reachable from PATH, or named by pythonBin. In an installed package the _env/.venv leg does not exist, so the interpreter is the python3 on your PATH; whichever one is chosen is logged at load time, and a missing Scrapling reports an error carrying the install line to copy.

pip install "scrapling[rag]>=0.5,<0.6"
python -m patchright install chromium

• Why `rag` and not `fetchers`: rag is fetchers plus markdownify, and the default extractionType is markdown — with plain fetchers the first fetch raises ModuleNotFoundError at the Markdown conversion.
• Version range: developed and verified against Scrapling 0.5, through interfaces that are not stable API surface (Selector, Spider, SessionManager, and the converter's noise-stripping helpers). The upper bound is pinned below 0.6; outside the range the guards still run, but the anti-injection cleaning and the crawl path may not behave as documented. In 0.5 the browser backend is patchright only (the old standalone stealth fetch is gone; its anti-detection arguments live on scrapling_render), capture_xhr became session-level recording (hence scrapling_session_network), and extra_flags became additive — this package sends none, and WebRTC leak protection is carried by the blockWebrtc switch.
• Browser engines are entirely your responsibility. The plugin downloads nothing. Without a browser it simply does not register the browser-dependent tools, so the model never wastes a turn calling them.
• **No @deepseek-ai/* under dependencies, by contract** (test/publish-manifest.test.ts enforces it). Consumer profiles install with nodeLinker: hoisted and autoInstallPeers: false, so a host package there becomes a real second copy whose module-level state splits from the running host's — every native tool call then throws while build and tests stay green. The host packages the artifact imports for value are required peers and stay externalized; the gate imports the built artifact and asserts that no host package body is inlined.
• The artifact is ESM-only and never committed — prepack rebuilds it, so what ships is always what the current source produces.

Shared version floor

If your profile pins @jayyuen66/dsh-plugin-shared with a pnpm overrides entry, that pin must be at least the version this package declares. A lower pin installs a copy without the names the sources use, and the host then reports N entries did not activate together with does not provide an export named ... at boot - the package installs fine and never loads, and no per-repository CI run can see it.

Verify it is live

1. The layer is mounted. dsh --profile web --dump-config shows this package's layer line.

• That command is the only accepted "it is in the layer stack" signal, and it is not read-only — it rewrites the profile's root assembly file. Read layer lines from stdout and check stderr separately: an unmatched patch target appears only on stderr, so the exit code alone proves nothing. Accept both # == and # == , patched by .

1. The startup log names the interpreter and the plan. One line prints the probed capabilities and how many of the tools registered; the chosen interpreter path is logged at load. An interpreter that starts but lacks Scrapling is named in the log together with the pip line to copy.
2. The settings card renders. The plugin detail page carries the scrapling card with every user-editable field grouped by purpose; toggling a value enables Save, and the write returns 200.
3. A tool round-trip. Ask the model to fetch a JavaScript-heavy page: the result row renders in the chat through the official tool-call view. On a machine without a browser the render / capture_xhr tools are absent from the model's tool list and their prompt guidance is gone too — that absence is the conditional registration working, not a failure.

What it does not provide

• It is not a search engine. Scrapling only sets a Google referer header for its google_search option; discovery needs a deployment-level searchEndpoint (see Settings). Without one the search provider stays unavailable and the built-in web_search keeps its behaviour.
• It downloads nothing. The Python interpreter and the browser engine are supplied by you; missing pieces remove the matching tools instead of being fetched.
• It opens no HTTP route of its own. The settings card is a host config form; there is no /api surface from this package.
• It adds no CLI subcommand.

Enabling it in dsh

• dsh.bundle.patch points at cordis.patch.yml, whose single registration line is - id: scrapling with the bare package name — the host scans dsh.client on bare package-name entries only.
• The client half registers two surfaces: the settings card (on the plugins.bundle.config slot, keyed by the bundle package name — the host dispatches entryKey: pkg.name, so the card renders on this package's detail page) and one result row per tool on tool.call.toolview, drawn with the official tool-call view props.
• The settings namespace is the entry id scrapling; precedence is settings-card value > registration-line config: > schema default. Setting a field back to its default saves a clear-override, not a write of the same value, so a later default change still reaches you.
• Gate policy and the sidecar environment are snapshotted once when the plugin is applied — settings changes take effect on the next start.
• Disabling a tool also removes its system-prompt guidance: the guidance is re-read at each prompt assembly against the tools that scope can actually see, so a machine without a browser loses the render / capture_xhr lines and a sidecar that never came up loses all of them. The section is placed next to the built-in web_fetch guidance.
• scrapling_crawl registers its own job kind, so job_list labels its jobs as crawls rather than shell jobs.

The tools

Thirteen tools, each behind one settings switch, all exercised by the test suite:

Tool · Backend · Needs a browser · Concurrency scrapling_fetch · Fetcher (curl_cffi) · no · concurrent; write verbs are serialised once allowWriteMethods is on scrapling_extract · Selector + CSS/XPath + adaptive · no · concurrent scrapling_render · StealthySession (patchright) · yes · serialised; the anti-detection switches (solveCloudflare / hideCanvas / blockWebrtc) live here scrapling_capture_xhr · one throwaway session, page-level recording · yes · serialised scrapling_session_open / scrapling_session_fetch / scrapling_session_network / scrapling_session_state_save / scrapling_session_state_load / scrapling_session_list / scrapling_session_close · Scrapling session classes · browser sessions only · serialised scrapling_crawl · Scrapling Spider framework, runs as a job · no · serialised scrapling_answer · any fetch path plus ctx.llm · with render: true · serialised

• Quotas. At most 8 open sessions (3 of them browser sessions, each a real Chromium process), 2 crawls in flight, per-page bodies clipped by *MaxOutputChars, 8 MB per protocol frame, 16 queued frames. A ceiling answers with a specific code (SESSION_LIMIT / CRAWL_BUSY / BAD_FRAME) the model can back off from.
• `scrapling_crawl` returns a job id immediately — a crawl takes minutes and a tool call has a timeout. Watch it with job_list, read it with job_output, stop it with job_kill; the tool needs a jobs registry that can attach a controller, and a profile without one simply does not register it (single-page fetching is unaffected). The crawl has a wall-clock ceiling (crawlMaxRuntimeSeconds) and never leaves the start domain. Politeness defaults follow Scrapling's spider defaults — robots.txt is not obeyed, per-domain delay is 0, up to 4 requests run in parallel — and is a deployment decision, not a per-call one (see Settings).
• `scrapling_answer` fetches the page first, then hands the text plus your question to a model and returns the conclusion instead of the whole page. It follows the model the current session is already using by default (read from the session's live request header, so a mid-session model switch is picked up), which burns no separate quota; pin answerProvider / answerModel to route page-reading elsewhere. Page text is untrusted input — the prompt frames it explicitly as data, but that defence is prompt-level: it lowers the risk, it does not remove it.

Render or static

scrapling_render is much slower and heavier than scrapling_fetch, so start static. Two criteria:

• If the rendered result is shorter than the static fetch, go back to static. Plenty of sites ship readable text from the server and then have their own script empty those containers and swap in canvas / Lottie animation — for those pages rendering can only lose content, and no wait setting brings it back. That is site behaviour, not a plugin defect.
• `waitSelector` waits for an element to appear, not for content to settle. The element can be removed after it matched, so "the selector matched" does not mean the content is in the result.

One thing that is not a criterion but worth knowing: axis ticks inside SVG cannot be fetched at all — the Markdown conversion emits no ` / nodes, so chart numbers are out of reach whichever tool you use. For actual values, go through scrapling_capture_xhr` to locate the JSON endpoint.

Replacing the built-in fetch and search

The plugin registers itself as a fetch provider and a search provider on ctx.web, so the built-in web_fetch / web_search tools actually run on Scrapling instead of dsh's plain-HTTP implementation. The model still calls web_fetch / web_search — tool names cannot be swapped (same-name registration throws), the provider seam is the supported channel. Both providers are registered by default (provideWebFetch / provideWebSearch), but that alone displaces nothing:

1. leave provideWebFetch / provideWebSearch at their default (true);
2. set fetchProvider / searchProvider to "scrapling" in the deployment config.

A configured id wins over auto-selection, which is why step 2 is required — and with two usable providers and no id named, ctx.web answers WEB_PROVIDER_AMBIGUOUS and the tool becomes unavailable outright. Two consequences of pointing that line at this plugin:

• If the sidecar cannot start, the built-in `web_fetch` reports `WEB_PROVIDER_CONFIGURED_UNAVAILABLE` rather than quietly falling back to plain HTTP.
• If you later remove the plugin but leave the line in, the built-in tools report WEB_PROVIDER_CONFIGURED_MISSING — go back and remove fetchProvider / searchProvider too.
• The fetch provider really applies the output budget: a body over fetchMaxOutputChars is cut to a head/tail window and flagged truncated, not passed through with a flag. Failures keep the sidecar code prefix (URL_BLOCKED, SIDECAR_TIMEOUT, SIDECAR_UNAVAILABLE), so a policy refusal is distinguishable from a crash or a timeout.

Search is split in two, because Scrapling is not a search engine: discovery goes to a deployment-level searchEndpoint, expected to be SearXNG-compatible JSON (GET ?q=&format=json returning {"results": [{"url", "title", "content"}]} — any backend producing that shape works); reading is this plugin's — the first searchRenderTopN results are actually fetched and, with a browser, rendered, so one web_search call returns both which pages matter and what they say. Every returned URL goes through the same gate as a fetch; a page that fails to read is skipped alone, because search results routinely contain dead links.

One consequence to know: dsh-web's fetch body is a closed union of html | text with no markdown kind, and adding one is a cross-package change a plugin cannot make. The fetch provider therefore reports Scrapling's Markdown as text, which the built-in renderer displays normally. Use scrapling_fetch directly when you need html or a CSS selector.

Settings

Settings live on the scrapling card in the plugin detail page, grouped by purpose, each field with a one-line "what changes if you edit this". The card's language follows the host locale. Values marked below live on the card; the deployment table after them is only settable through the config: block of cordis.patch.yml.

Field · Type · Default · Purpose pythonBin · string · "" · Interpreter name or absolute path. Empty = the repository _env/.venv when executable, otherwise python3 on PATH failFastOnMissingPython · boolean · true · Refuses to load when the sidecar process cannot start (wrong path, protocol mismatch, handshake timeout). An interpreter that starts but lacks Scrapling always degrades to "tools unavailable" with the path named in the log, whatever this switch says fetchEnabled / extractEnabled / renderEnabled / captureXhrEnabled / networkEnabled / sessionEnabled / sessionStateEnabled / crawlEnabled / answerEnabled · boolean · true · One kill switch per tool or tool pair. sessionStateEnabled gates the state save / load pair, which also needs a deployment-side sessionStateDir extractionType · enum · markdown · markdown / html / text; all three run the same anti-injection cleaning, and html additionally strips event-handler attributes, so it is not byte-for-byte raw HTML mainContentOnly · boolean · true · Extraction scope *MaxOutputChars · natural · fetch / extract / render 200000, xhr 120000 · Per-tool output caps. The sidecar clips by characters, the host then trims to a byte budget crawlMaxItems · natural · 1000 · Ceiling over the model-supplied maxPages. An out-of-range maxPages is passed through so the sidecar reports the parameter error rather than this plugin silently clamping it stripInlineImages · boolean · true · Drops inline data: URI images from the body, keeping only alt text — pure token noise for the model. Code fences are left untouched; pages without inline images come back byte-identical. Stripping happens before the cap. Turn off only when the raw base64 is the data (colours, fingerprints) headless / networkIdle · boolean · true / false · Browser mode and whether a page must reach network idle before it is read blockAds · boolean · true · Stacks on top of the SSRF guard rather than competing with it: the guard is registered last (so the engine runs it first) and yields with route.fallback() pierceShadow · boolean · false · Merges open shadow roots into the body — the switch for pages that hide their text in shadow DOM. The model can also override it per call on the browser tools waitSelectorState · enum · attached · The state a waitSelector element must reach (attached / detached / hidden / visible) sidecarHandshakeTimeoutMs / fetchTimeoutMs / sidecarGraceMs · natural · 30000 / 120000 / 5000 · Handshake budget, per-call cooperative timeout (declared on the tools' timeoutMs), and how long the client waits for the process to exit before forcing it down. scrapling_crawl passes no timeout — a crawl is measured in minutes dataDir · string · "" · Sidecar working directory (the adaptive fingerprint database lands here); empty = ~/.dsh-scrapling. If it cannot be created, failFastOnMissingPython decides whether the plugin refuses to load or degrades to no tools provideWebFetch / provideWebSearch · boolean · true · Whether the ctx.web providers are registered at all — see Replacing the built-in fetch and search answerProvider / answerModel / answerMaxTokens · string / natural · "" / "" / 2000 · The model scrapling_answer routes to and the per-reply token cap. Provider and model left empty mean "follow the model the current session is already using" allowedHosts · string · "" · Extra host names that skip the public-address check (comma-separated). Pointing it at an intranet address means giving up SSRF protection for those targets on purpose syntheticDnsRanges · string · "" · Manual fallback for synthetic DNS: a comma-separated CIDR list of your proxy's fake-IP pool, for resolvers that mix synthetic and real answers. Invalid entries are dropped — a broken fallback degrades to "stay strict", not to a crash | proxyUrl | string | "" | Where fe