dsh-web-fetch

dsh-web-fetch

Dual-source web content fetcher for DeepSeek Harness — CDP browser rendering + Tavily Extract, each as a standalone LLM tool.

English | 中文

Why dsh-web-fetch?

DeepSeek Harness's ctx.web.registerSearchProvider throws WEB_PROVIDER_AMBIGUOUS when multiple providers are registered for the same capability — the model can't choose.

dsh-web-fetch takes a different path: each data source is a standalone DSH tool (web_fetch_cdp / web_fetch_tavily) with its own description + schema. The LLM picks the right tool based on context, not hard-coded rules.

Both can be independently enabled/disabled — disabled tools are hidden from the LLM entirely.

Parameters (identical for both tools):

Renamed in v0.1.5-rc.1: this parameter used to be named query. That name collided with Tavily's own /extract request field query (a search term), and it invited callers to pass a search topic where a URL was required. It is now url. The old name is deliberately not accepted as an alias — it fails loudly with invalid arguments: missing required property "url", which is itself the migration hint.

Features

• Dual pluggable strategies — CDP + Tavily out of the box, add a new one with 1 file + 1 line
• Zero-intrusion — Cordis bundle plugin, no DSH core patch. All via ctx.tools.register plus an exported volatile Config schema (dsh ≥ 0.1.7: the settings namespace is the cordis row id, and edits apply live — installSection/settingsScope no longer exist)
• Zero extra deps — only @deepseek-ai/dsh-settings, @deepseek-ai/dsh-tools, @deepseek-ai/schemastery
• Live config — settings UI card + ~/.dsh/settings.yaml hot-reload, no restart needed
• Concurrency-safe — isConcurrencySafe: true, supports AbortSignal

Architecture

dsh-web-fetch/
├── cordis.patch.yml        # bundle patch (install/uninstall)
├── package.json            # dsh.client injection
├── src/
│   ├── index.js            # Cordis apply() + 2 tool registrations
│   ├── types.js            # FetchStrategy契约 (JSDoc)
│   ├── helpers.js          # withTimeout / plainText / truncate
│   └── strategies/
│       ├── cdp.js          # CDP: raw http + manual WS frames, no `ws` dep
│       └── tavily.js       # Tavily Extract API
├── lib/client.js           # Plugins-page settings card (React + locale zh/en)
└── tests/
    ├── entry.test.mjs           # host entry + manifest + engines table + key parity
    ├── host-integration.test.mjs # real Cordis + ToolRuntime + file settings provider
    ├── test-cdp-unit.mjs        # 21 tests (helpers 8 + factory 5 + router 8)
    ├── test-cdp-frames.mjs      # 5 tests (RFC6455 frame encode/decode)
    ├── test-tavily-unit.mjs     # 13 tests (availability, parsing, error paths, malformed payloads)
    └── client-smoke.mjs         # 15 checks (browser half: render + interaction + read-only)

Strategy contract (src/types.js):

// New source = implement this, then register once in src/index.js
export function makeMyStrategy(config) {
  return {
    id: "my",
    title: "My Fetcher",
    available() { return Boolean(config.apiKey) }, // cheap check, no I/O
    async fetch(req, signal) {
      return { sources: [{ url, title, snippet, content, provider: "my" }], truncated: false }
    },
  }
}

Quick Start

1. Install

# from source
git clone https://github.com/runfali/dsh-web-fetch.git && cd dsh-web-fetch
pnpm install          # DSH loader resolves deps from plugin dir, not host

dsh plugin --profile web add ./dsh-web-fetch
# or after publishing:
# dsh plugin --profile web add dsh-web-fetch

# restart DSH
# systemd: sudo systemctl restart dsh

Why pnpm install? Cordis plugin loader resolves imports from the plugin directory only. Without it you'll get ERR_MODULE_NOT_FOUND: @deepseek-ai/schemastery.

2. Configure (UI recommended)

Open the Web UI's Plugins → 通用 Web 内容获取（web-fetch）

• Enable toggles — CDP / Tavily checkboxes at the top
• CDP group — CDP Endpoint (default http://10.200.0.5:9222), Timeout ms (60000), Extra wait after load ms (2000)
• Tavily group — Endpoint (https://api.tavily.com/extract), API Key (leave empty to disable), Timeout ms (30000)

Saving writes the web-fetch entry of the profile patch and applies live (the fields are declared volatile, so no restart and no fiber remount).

- id: web-fetch
  config:
    cdpEnabled: true
    cdpEndpoint: 'http://10.200.0.5:9222'
    cdpTimeoutMs: 60000
    cdpWaitMs: 2000
    tavilyEnabled: false
    tavilyEndpoint: 'https://api.tavily.com/extract'
    tavilyApiKey: ''
    tavilyTimeoutMs: 30000

3. Use

The LLM will see the tools automatically. Manual test:

User: 用 web_fetch_tavily 提取 https://example.com 的正文
User: 用 web_fetch_cdp 抓取 https://example.com 这个需要渲染的页面

Tool output shape:

{
  "sources": [{ "url": "...", "title": "...", "snippet": "...", "content": "...", "provider": "cdp|tavily" }],
  "truncated": false
}

Adding a New Data Source

1. Create src/strategies/my.js implementing FetchStrategy
2. Register in src/index.js:

import { makeMyStrategy } from "./strategies/my.js"
ctx.tools.register(makeToolDef("my", makeMyStrategy, "myEnabled", current))

3. (Optional) Add fields to Config + FIELD_VIEWS in lib/client.js

No changes to router or core logic needed.

Development

pnpm test          # full suite
pnpm test:host     # real-host contract tests only

All tests are offline (no real browser, no real Tavily call). tests/host-integration.test.mjs drives the plugin on the real dsh objects (@deepseek-ai/cordis, dsh-tools, dsh-system-prompt, dsh-settings-file) rather than hand-rolled stubs, and skips loudly when those packages are unresolvable.

Requirements

Why the disjunction? npm semver satisfies a prerelease only from a range group that itself carries a prerelease with the same [major, minor, patch] tuple. The previous single >=0.1.2-alpha.3 <0.2.0 group therefore did not cover 0.1.5-rc.1 — "claims 0.1.5 support but fails its own claim". The same trap applies to every dependency range on a dsh package. tests/entry.test.mjs pins this with a 13-row decision table, a counter-proof against the old range, and a line-by-line cross-check against the host's real semver.satisfies.

Limitations & Notes

• tavilyApiKey is stored as plain text in the settings document (settings system, not credential vault). For sensitive envs, override via profile cordis.patch.yml.
• CDP uses Node's native http + hand-rolled WebSocket frames (no ws dep). permessage-deflate is not used — compatible with cloakbrowser default (compression off).
• Version compatibility: see the Requirements table above (machine-readable in package.json under dsh.engines.dsh).

Contributing

PRs welcome! Please:

1. Keep zero extra deps
2. Add a strategy under src/strategies/ with unit tests
3. Update both zh/en locales in lib/client.js if adding settings

dsh 0.2.1-rc.2 适配结论

对桌面端 D:\DeepSeek Harness\（FileVersion 0.2.1-rc.2）做了 asar 解包源码比对 + 宿主真闸实跑：

• 契约点零漂移：dsh-tools 的 defineTool、dsh-settings 的 configure()/volatileForm、 dsh-web 的 registerSearchProvider、client 的 configForms.whileServed 与 slots.inject 在 0.2.1 全部保持原形状（官方 dsh-client-ui-settings-web-search 自己也仍 require("@deepseek-ai/dsh-client-ui-primitives")）。src/ 与 lib/client.js 未改一行。
• 本轮补的是 pnpm 安装期，不是加载闸：旧四段区间在真闸下 evaluatePluginCompatibility(...) → undefined（PASS）， 即 0.2.1 下本来就能加载；被拒的是严格模式（semver.satisfies 默认选项，pnpm 安装期）。 真负例 >=0.1.7-alpha.0 <0.1.8 被判 REJECT，证明闸确实在判。
• 闸语义在 0.2.1 变了：unmetRequirements 现在「顶层 engines.dsh 存在时它独自判定、peers 被忽略」。 本仓区间写在嵌套的 dsh.engines.dsh（不是顶层），故实际仍由 peers 判定——真闸实测确认。
• 两个 semver 模式并存：测试改成两列（严格 / 宿主闸 includePrerelease:true），三处分叉行 （0.1.8-rc.1 / 0.2.2-rc.1 / 0.3.0-alpha.0）显式钉住，避免把 <0.3.0 误读成「0.3 全拦」。
• 验证：npm test 17 + 7 组 + cdp/tavily/frames/client 全绿，0 skipped；host-integration 跑在 真实的 dsh-tools/dsh-settings/dsh-system-prompt@0.2.1-rc.2 上。

dsh 0.2.0-rc.1 适配结论

对桌面端 D:\DeepSeek Harness\（FileVersion 0.2.0-rc.1）做了 asar 解包源码比对 + 真机闸实测。要点：

• 唯一必改项是兼容区间：原区间在 0.2.0-rc.1 下被启动闸拒绝 （dsh: skipping profile bundle ...），插件整个 bundle 不加载（web 与 desktop 同时失效）。 追加 || >=0.2.0-alpha.0 <0.3.0 后放行。
• 闸只读 peerDependencies：判定函数（dsh-app-boot 的 evaluatePluginCompatibility）只遍历 peerDependencies 里 @deepseek-ai/dsh* 的条目， 从不读 dsh.engines.dsh（全树 grep 零消费者）。两者必须逐字一致，测试已守护。
• 两种 semver 模式：宿主闸用 includePrerelease: true，此时「预发布可见性」规则被绕过， 于是上界自身的预发布也被放行（<0.1.8 放行 0.1.8-rc.1、<0.3.0 放行 0.3.0-alpha.0）； 严格模式（pnpm 安装期）会拒绝它们。所以上界拦的是正式版，不是预发布。 若要连预发布一起拒，上界须写成 <0.3.0-0。
• 凭证细节：详细取证、改动清单、测试结果与诚实缺口见 docs/DSH-0.2.0-ADAPTATION.md。

License

MIT © 2025 dsh-web-fetch

中文说明

完整中文文档请见 README.zh-CN.md。

dsh-web-fetch 是 DeepSeek Harness 的通用网页内容获取插件，提供 双数据源、可插拔策略 能力。核心设计是规避 registerSearchProvider 的 WEB_PROVIDER_AMBIGUOUS 限制，将每个数据源注册为独立工具，由 LLM 自主决策。lib/client.js 内置完整中英文界面。