dsh-web-search-public

[!NOTE]

📦 This plugin lives in the dsh-plugins monorepo — file issues & pull requests there.

npm: @hy-sde-org/dsh-web-search-public

dsh-web-search-public — credential-free web search for DeepSeek Harness

A zero-API-key web search provider for DeepSeek Harness. With a single web_search call it fans the query out to all five public engines in parallel — Startpage, DuckDuckGo, Ecosia, Google, Mojeek — and consolidates the answers by cross-engine consensus, so no single engine's bot challenge, timeout, or empty page can degrade the search. No API keys, no accounts, no fetch provider, no setup.

Identity · Value Package · @hy-sde-org/dsh-web-search-public Plugin id · web-search-public Provider id · public Engines (fanned out) · startpage, duckduckgo, ecosia, google, mojeek

Based on oh-my-pi — the credential-free engine

scrapers are ported from oh-my-pi’s web/search/providers, and the provider is a faithful

port of oh-my-pi’s parallel searchPublicWeb aggregate (fan-out, soft/hard deadlines,

consensus merge), adapted to the DeepSeek Harness ctx.web seam.

oh-my-pi is MIT-licensed (Mario Zechner, Can Bölük); see

THIRD-PARTY-NOTICES.md.

Why

DeepSeek Harness ships search providers that need an API key (DeepSeek, Exa, Perplexity). This bundle is the credential-free default: each search is an anonymous request to every public search engine in parallel. A bot-challenged, rate-limited, or down engine is absorbed — the answer comes from whichever engines responded, and results that several engines agree on are ranked first. The whole call is bounded by soft/hard deadlines, so a hung engine can never pin a search to the tool budget.

Prerequisites

• Node.js 22.19 or newer with npm and pnpm on PATH;
• DeepSeek Harness 0.2.0-rc.2 including the standard dsh CLI;
• no API keys — nothing else.

Install the Harness CLI and pnpm before continuing:

npm install --global @deepseek-ai/dsh@0.2.0-rc.2 pnpm
dsh --version

Quick start

Route A — published npm package (recommended)

dsh plugin --profile web add @hy-sde-org/dsh-web-search-public

Route B — from source (validate this checkout or hack on the plugin)

git clone git@github.com:hy-sde/dsh-plugins.git
cd dsh-plugins/dsh-web-search-public
pnpm install
PACKAGE_TARBALL="$(pnpm pack | tail -n 1)"
dsh plugin --profile web add "$PWD/$PACKAGE_TARBALL"
cd ../..

pnpm pack runs the normal prepack build and produces a tarball containing dist/. A direct github: dependency does not contain built output and is not a supported install path — always install the built tarball (or the published package).

Verify the composed configuration

dsh web --dump-config

The composed tree must show web.config.searchProvider: public and a web-search-public plugin row loading @hy-sde-org/dsh-web-search-public.

Run

dsh web

Ask the agent to web_search something. The tool returns rendered sources from the consensus-merged fan-out.

Uninstall

dsh plugin --profile web remove @hy-sde-org/dsh-web-search-public

Already shipped? If your harness checkout already mounts web-search-public in its base

bundle (a recent release adopted this package as the shipped default), skip installation

— it is already the active provider. Adding this bundle on top would duplicate the loader

row and fail at boot (“duplicate loader entry id”).

What the bundle does

The package declares a DSH bundle (dsh.bundle.patch → cordis.patch.yml), so dsh plugin installs it and applies its patch layer to the profile:

1. sets web.config.searchProvider to public (making the credential-free fan-out the default),
2. inserts the web-search-public plugin row.

The keyed DeepSeek route (web-search-deepseek, provider id deepseek-official) is not disabled — deployments that prefer it keep their explicit choice by setting web.searchProvider: deepseek-official and providing the DEEPSEEK_API_KEY credential. No fetch provider is mounted and tool-web keeps fetch: false: the model can search, not fetch arbitrary URLs, which defers SSRF exposure entirely.

How the fan-out works

PublicSearchProvider.search() races engines concurrently (a faithful port of oh-my-pi’s searchPublicWeb):

┌────────────────────── fan-out (one query) ──────────────────────┐
         ▼            ▼            ▼            ▼            ▼
     startpage    duckduckgo     ecosia      google       mojeek
         └────────────► consensus merge ◄────────────────┘

• Every engine is asked in parallel. The race returns at the earliest of: all engines

settled, the soft deadline (5 s default) with at least one success in hand, or the hard deadline (30 s default) regardless — then aborts every still-running engine. If the soft deadline passes with no success yet and not everything has failed, the call keeps waiting up to the hard deadline for the first success.

• Sources are deduplicated across engines (case/www./trailing-slash-normalized URLs) and

ranked by cross-engine consensus — how many engines returned a URL — then best rank, then engine order. The most informative snippet wins; the earlier engine’s title/URL win equal-rank ties.

• Each engine still runs under its own AbortSignal.timeout(timeoutMs) race, so an engine

that ignores the aggregate’s cancellation dies at its own cap.

• Rate-limit resilience is best-effort, layered. A hard block (HTTP 4xx) or

a zero-result page repeated on consecutive searches opens *that engine’s* circuit breaker: it is skipped for engineBackoffMs (5 min default, doubling per consecutive trip up to maxEngineBackoffMs), with no request sent, and probed again after the window. Retryable transport failures (HTTP 5xx, timeout, network fetch failure) retry the whole fan-out after an exponential backoff (maxRetries/retryDelayMs, abort-aware sleep); hard blocks are never retried. Once retries are exhausted — or every engine is down — the provider fast-fails for failureCooldownMs (5 min default) with a retry in about Ns error instead of re-blasting engines that just throttled us (which would deepen the block). A pure all-no results aggregate is query-level and is never retried or cooled down.

• available() is true whenever at least one engine is configured — the provider is always

usable, which is the point.

• A caller abort (signal.aborted) aborts the fan-out and throws WEB_ABORTED.
• Only if every engine fails does the provider throw WEB_PROVIDER_ERROR with each

engine’s reason (id: ; …). Zero organic results, a timed-out engine, and a failed HTTP request all count as “this engine has nothing” and are absorbed by the merge.

The worst case for the default configuration is ≈ hardDeadlineMs (30 s), well under the 60-second search budget dsh-tool-web books in the shipped harness bundle.

Browser escalation (bot walls)

Google, Ecosia and Mojeek are browser-backed (an omp browserFetch port): each engine first tries a plain fetch carrying coherent Chrome navigation headers (client hints derived from its UA), and when that is answered with a bot wall — an enable-JS retry page, a Cloudflare managed challenge ("Ecosia Firewall"), Mojeek's ALTCHA proof-of-work wall, or Google's "unusual traffic" /sorry gate — it escalates to a real browser via ctx.browser.fetchPageHtml (@hy-sde-org/dsh-browser service, stealth launch or CloakBrowser patch backend): home-URL cookie seeding, best-effort ready-selector wait, and ALTCHA auto-solve (force-click of the shadow-DOM checkbox + PoW redirect wait, omp's solveCaptcha). The circuit breaker only opens when every transport — fetch *and* browser — is blocked.

Escalation is optional: mount the browser service row before this provider and the browser-backed engines light up; without it they stay fetch-only (original behavior), and timeoutMs (default 30 s) is the per-engine budget that covers fetch → browser steps.

Configuration

All options are optional; constants fill the defaults. Configure via the plugin row’s config: in your profile patch, or programmatically via ctx.plugin(...).

Option · Default · Purpose engines · startpage, duckduckgo, ecosia, google, mojeek · engine ids the fan-out races; this order is the tiebreak for consensus ties; unlisted engines stay disabled timeoutMs · 30000 · per-engine transport timeout (ms): bounds one engine even if it ignores aggregate cancellation; browser-backed engines may spend it on fetch → stealth-browser escalation, so the default is 30 s softDeadlineMs · 5000 · soft aggregate deadline — return as soon as all engines settled or this passes with ≥1 success hardDeadlineMs · 30000 · hard aggregate deadline — return whatever we have, even nothing; must be ≥ softDeadlineMs maxRetries · 1 · retries for the all-engines-failed aggregate when at least one engine died from a *retryable* transport failure (HTTP 5xx, a timeout, or a network fetch failure). Each retry re-runs the whole fan-out after a backoff, so a short engine throttle that strips the first attempt still yields results. 0 disables retrying. Hard blocks (HTTP 4xx) are never retried — they open that engine’s circuit breaker — and a pure all-no results aggregate is query-level and never retried. retryDelayMs · 2000 · base delay before retry #1 (ms); doubles on each subsequent attempt. failureCooldownMs · 300000 · fast-fail window (ms) after an all-engines-down failure (retry exhausted, or every engine hard-blocked/tripped): searches in this window return a clear retry in about Ns error instead of re-blasting engines that just throttled us (which would deepen the block). 0 disables the window. engineBackoffMs · 300000 · base per-engine circuit-breaker window (ms): an engine that hard-blocks (HTTP 4xx) or returns no results on consecutive searches is skipped for this long instead of being re-blasted on every search. Doubles per consecutive trip up to maxEngineBackoffMs; resets after a successful probe. 0 disables the per-engine breaker. maxEngineBackoffMs · 3600000 · cap for the doubled per-engine circuit-breaker window (ms). userAgent · browser-shaped constant · User-Agent sent to the engines (deliberately not the product UA)

- id: web-search-public
  name: '@hy-sde-org/dsh-web-search-public'
  config:
    engines: [startpage, duckduckgo, google, mojeek]   # drop Ecosia, reorder
    timeoutMs: 30000
    softDeadlineMs: 4000
    hardDeadlineMs: 25000
    maxRetries: 1
    retryDelayMs: 2000
    failureCooldownMs: 300000
    engineBackoffMs: 300000
    maxEngineBackoffMs: 3600000

PublicEngineId values are the exported engine ids; any id outside the known set is dropped at startup, and a zero-engine config makes available() false (surfacing WEB_PROVIDER_CONFIGURED_UNAVAILABLE).

Errors

Search failures use the web seam’s WEB_* codes:

Code · Meaning WEB_PROVIDER_ERROR · every engine failed; message lists id: reason; …

A rate-limited all-failed aggregate (transport signature, retries exhausted) surfaces the same WEB_PROVIDER_ERROR with public search engines are rate limited (…); retry in about Ns. WEB_ABORTED · caller aborted mid-search WEB_PROVIDER_CONFIGURED_UNAVAILABLE · configured provider not available WEB_PROVIDER_CONFIGURED_MISSING · configured provider id not registered

Privacy and security notes

• No credentials. The provider never reads a key, so a leak cannot happen. Engines

receive the query, the browser-shaped User-Agent, and nothing else.

• Treat queries as public. Each query is sent to public search engines as an anonymous

request — including, because of the fan-out, several engines at once. Do not search for secrets or PII you would not paste into a public search box.

• Amplification is inherent. Every search contacts every configured engine in parallel

(up to 5 anonymous scrapes per query). That is the price of resilience and consensus, and it raises bot-challenge / rate-limit exposure compared with a single-engine chain — see docs/operations.md.

• No fetch provider. tool-web fetch: false stays: the model cannot be pointed at

arbitrary URLs, so SSRF and unsafe-content surfaces stay closed.

• Redirect policy. All engine fetches use redirect: "error" — redirects are treated as

failures and absorbed by the merge rather than leak the caller onto an external location.

• Challenges are expected. Startpage, Ecosia, and Google frequently serve bot checks;