dsh-web-gateway

📦 @goodandready/dsh-web-gateway

---

Resilient web search and page extract tools for DeepSeek Harness.

When the built-in web_search / web_fetch tools hit rate limits, captchas, or empty results, this plugin gives the model a separate fallback chain with different tool names (no registration collisions).

Free tiers are tried before keys: Firecrawl keyless, Keenable keyless, local SearXNG, and DuckDuckGo run before any keyed provider. Every result names the exact mode (firecrawl:free, firecrawl:key#2).

Overview / The Problem

DSH ships web_search and web_fetch, but a single upstream can rate-limit, captcha, or return an empty result. This plugin adds separate web_gateway_* tools with an explicit attempt model: each provider expands into free and keyed attempts, every free attempt runs before any keyed attempt, and each skip or failure is reported with its mode.

The plugin does not replace the built-in tools and does not register web_search / web_fetch. It is a fallback surface for the model, with operator-controlled provider order, caps, credentials, and diagnostics.

Architecture

graph LR
  A[Agent tool call] --> B[web_gateway_search / extract / research]
  B --> C[Attempt expansion: provider + mode]
  C --> D[Free tier: keyless Firecrawl, Keenable, SearXNG, DuckDuckGo, Jina, readability]
  D --> E[Keyed tier: Firecrawl, Keenable, Tavily, Brave, Exa, Crawl4AI]
  E --> F[Guards: per-mode caps, circuit breaker, telemetry]
  F --> G[Result, skipped reasons, cache]
  B --> H[URL policy: public IP, same-origin redirects, bounded bodies]

Tools

HTTP API

The settings card uses two same-origin routes. Loopback requests are always allowed; remote browsers are allowed only when Sec-Fetch-Site: same-origin or when Origin and Host match the request host. This lets the card work through a LAN/Tailscale/reverse-proxy DSH UI without opening the routes to cross-site traffic.

A provider is skipped on missing key, empty result, HTTP 429/5xx, rate limits or network error. Built-in Circuit Breaker temporarily pauses providers suffering consecutive failures to avoid long timeouts. Results are cached in memory (configurable TTL). Successful responses include provider, skipped (providers not used and why), and cached.

Free tier first

The plugin expands each provider into free and keyed attempts. With freeTierFirst (default), every free attempt runs before every keyed attempt:

1. Firecrawl keyless
2. Keenable keyless
3. SearXNG (if configured)
4. DuckDuckGo
5. Firecrawl key
6. Keenable key
7. Tavily
8. Brave
9. Exa

freeTierFirst=false interleaves free and keyed attempts per provider. To disable keyless entirely, set firecrawlKeylessSearch=false, firecrawlKeylessFetch=false, and keenableEnabled=false.

Keenable joins the default order on upgrade; set keenableEnabled=false to opt out.

A provider with a comma-separated credential value uses its keys in order. Authentication, rate-limit, and quota failures rotate to the next key.

Extract URLs are validated: http/https only, no embedded credentials, and hostnames must resolve to public IPs (loopback / RFC1918 / link-local / metadata blocked). The public-IP check is repeated at connect time by the undici dispatcher. Set allowInternalUrls only if you intentionally need internal targets; even then cloud providers are skipped and only local readability runs.

Install

dsh plugin --profile web add @goodandready/dsh-web-gateway

Restart the web profile after install.

Credentials

Add keys under Settings → Credentials (or $DSH_HOME/.credentials.yaml). The settings card stores only credential names, never secret values:

A credential value may contain multiple keys separated by commas; they are tried in order.

Settings

Configure in Settings → Plugins → Web Gateway:

Limitations

• Local readability rejects PDF and DOCX with an explicit reason; Jina or Firecrawl are the document parsers. Do not expect local binary-document conversion.
• Keyless Firecrawl and Keenable receive public queries/URLs without an API key. Disable them if that egress is not acceptable.
• The settings card and its routes are same-origin; a reverse proxy must preserve Origin / Host (or Sec-Fetch-Site) for remote browsers.
• Screenshots are optional, requested only with take_screenshot, and capped at 5 MB per file.
• A provider timeout counts as a failure for the circuit breaker; repeated timeouts eventually pause that provider.

How it relates to core DSH web tools

DeepSeek Harness already ships web_search and web_fetch via @deepseek-ai/dsh-tool-web. This plugin does not replace them. It registers web_gateway_* tools so the model can fall back when the built-in path fails.

Development

npm test
npm run check

Tests run offline with mocked fetch (no live network).

Visual verification

Plugins settings card for Web Gateway — Dark and Light themes side by side:

License

MIT © GooDAnDReaDY