pivot-web-search

Pivot Web Search

Resilient multi-provider web search and page extraction for MCP hosts, the command line, and host adapters.

What Is This?

Pivot Web Search exposes one set of shared search, fetch, and configuration services through three interfaces:

• MCP server (pivot-web-search-mcp) — for Claude Code and any MCP-aware host
• CLI (pivot-web-search) — for people and shell scripts, no AI host required
• JSON bridge (pivot-web-search-bridge) — a host-neutral subprocess interface for adapters

It routes each query across multiple providers, fails over automatically when results are weak or a provider is down, tracks quota and health, and extracts page content locally — with no Anthropic API dependency, so it works on Amazon Bedrock and other API providers. A Claude Code plugin and an optional DeepSeek Harness adapter provide host-specific installation; adopting Pivot never requires patching a host's source.

DuckDuckGo works with no API key; adding an API-backed provider (Tavily, Brave, Gemini) improves reliability.

Choose Your Interface

You use… · Start here Claude Code · Install the plugin Codex · Install the plugin Another MCP host (Cursor, Claude Desktop, …) · Configure pivot-web-search-mcp Terminal / scripts / CI · Install and run the CLI DeepSeek Harness · Install runtime + DSH bundle

Key Features

• One runtime, three interfaces — the same search/fetch/config services back MCP, the CLI, and the JSON bridge.
• Multi-provider failover — quota-aware routing across providers; weak or exhausted providers are skipped, partial results kept as fallback.
• Super mode — query all providers in parallel, dedupe by URL, rank by cross-provider agreement.
• Local page extraction — full content via trafilatura, with Next.js/Nuxt.js SPA fallbacks and an optional JS renderer.
• Hot-reloadable config — add/remove/reorder providers and proxies via YAML; changes apply on the next request.
• Actionable diagnostics — when everything fails, get per-provider failure reasons and suggestions, not a generic error.

Single-provider search fails when an API is unavailable, rate-limited, blocked, or out of quota. Pivot routes each request across configured providers, keeps usable partial results, and exposes the same behavior through MCP, CLI, and adapter-friendly JSON.

Prerequisites

• uv — manages Python and dependencies automatically (used by every interface).
• At least one search provider (DDG needs no API key).

Recommended: configure at least one free API key — Tavily (1000 credits/month, no card) or Brave (1000 queries/month, card required). DDG is a free fallback but can be unreliable under heavy use.

Per-host setup is covered in each Quick Start below.

Quick Starts

Claude Code plugin

Step 1 — add the marketplace (one-time):

claude plugin marketplace add https://github.com/Tannnnhauser/pivot-web-search.git

Step 2 — install:

claude plugin install pivot-web-search

The plugin prompts for configuration at install time:

Setting · Description Tavily API Key · Stored in system keychain. Set to enable Tavily (or inherit TAVILY_API_KEY from the shell). Brave Search API Key · Stored in system keychain. Set to enable Brave (or inherit BRAVE_API_KEY). Gemini API Key · Stored in system keychain. Set to enable Gemini (or inherit GEMINI_SEARCH_API_KEY / GOOGLE_STUDIO_API_KEY). Proxy URLs · Comma-separated proxies to try in order. direct is always appended as the final fallback — to disable that, use ~/.pivot-web-search/proxies.yaml.

Providers are enabled automatically for whichever keys you supply; DDG is always on. Routing order and timeouts come from smart defaults — the order you supply keys does not matter. Reconfigure anytime via claude plugin configure pivot-web-search.

Verify: ask Claude Code to run WebSearchConfig with action status — you should see a provider health report.

Key resolution & macOS GUI caveat

For each provider key the plugin reads (1) the standard env var (e.g. TAVILY_API_KEY) inherited from the shell — wins if set — then (2) the /plugin UI value (injected as PIVOT_USERCONFIG_TAVILY_API_KEY). To use the UI value instead, unset TAVILY_API_KEY in your shell.

macOS: when Claude Code launches from Spotlight or the Dock it does not see ~/.zshrc exports. Start it from a terminal, set keys via the /plugin UI, or add them to the env block in ~/.claude/config.json.

Codex plugin

Add the same GitHub marketplace and install the same plugin payload:

codex plugin marketplace add Tannnnhauser/pivot-web-search --ref main
codex plugin add pivot-web-search@pivot-web-search-plugin

Start a new Codex chat after installation. Refresh future releases with:

codex plugin marketplace upgrade pivot-web-search-plugin

Codex reads provider keys from the standard environment variables listed above. DDG remains available without a key; advanced provider and proxy configuration under ~/.pivot-web-search/ is shared with every host.

MCP hosts

Any MCP-aware host (Claude Code, Claude Desktop, Cursor, …) can run the server directly via uvx — no clone, no venv. Add to your .mcp.json:

{
  "mcpServers": {
    "pivot-web-search": {
      "command": "uvx",
      "args": [
        "git+https://github.com/Tannnnhauser/pivot-web-search.git#subdirectory=plugins/pivot-web-search"
      ],
      "env": {
        "TAVILY_API_KEY": "tvly-...",
        "BRAVE_API_KEY": "BSA...",
        "GEMINI_SEARCH_API_KEY": "AI..."
      }
    }
  }
}

Pin a version with @v1.1.0:

git+https://github.com/Tannnnhauser/pivot-web-search.git@v1.1.0#subdirectory=plugins/pivot-web-search

Advanced provider/proxy config lives in ~/.pivot-web-search/*.yaml (see Configuration) and applies to every launch method.

CLI

The installed package provides pivot-web-search, a human-facing command backed by the same services as the MCP tools — for use in a terminal, a script, or CI, without going through an AI host. Install it standalone with uv:

uv tool install 'git+https://github.com/Tannnnhauser/pivot-web-search.git@v1.1.0#subdirectory=plugins/pivot-web-search'

pivot-web-search search "latest Python release" --format json
pivot-web-search fetch https://example.com --format md
pivot-web-search config status

`search` flags:

Flag · Default · Description --max-results · 5 · Number of results (1–10, or 1–20 with --super) --provider · auto · Force a configured provider by name --super · off · Query all providers in parallel --news · off · Search news instead of the general web --timelimit · — · d / w / m / y recency filter --include-answer · off · Include an AI answer when supported --search-depth · basic · basic or advanced (Tavily) --topic · general · general or news (Tavily) --days · — · Limit news to recent N days --include-domains · — · Domain allowlist --exclude-domains · — · Domain blocklist --include-content · off · Return pre-extracted page content (Brave LLM Context) --max-content-tokens · 8192 · Token budget for --include-content --region · wt-wt · DDG region (CLI-only; no MCP equivalent) --format · md · md or json

`fetch` (alias extract): --query (relevance hint for JS renderers), --max-chars (per-URL truncation), --format (json default, or md).

`config`: positional action, status (default) or reload.

DeepSeek Harness

The optional pivot-web-search-dsh Profile Bundle registers Pivot as DSH's existing web_search and web_fetch provider through DSH's published Bundle, ctx.web, and ctx.subprocess APIs. The model keeps seeing DSH's standard tools — no second set of Pivot-specific tools. Host integrations use public extension APIs; the adapter requires no DSH fork or source patch.

Install the runtime and the Bundle:

uv tool install 'git+https://github.com/Tannnnhauser/pivot-web-search.git@v1.1.0#subdirectory=plugins/pivot-web-search'
dsh plugin --profile web add pivot-web-search-dsh

Restart the profile after adding the Bundle. It selects provider pivot as searchProvider/fetchProvider and enables DSH's tool-web entry (which the shipped web profile disables). Verify with dsh --profile web --dump-config — you should see searchProvider: pivot, fetchProvider: pivot, an enabled tool-web, and pivot-web-search-provider.

Provider keys are forwarded explicitly by cordis.patch.yml (TAVILY_API_KEY, BRAVE_API_KEY, GEMINI_SEARCH_API_KEY, GOOGLE_STUDIO_API_KEY, plus PIVOT_WEB_SEARCH_PROXIES); config files under ~/.pivot-web-search/ are read directly.

Custom providers: the bridge subprocess receives only the env vars listed in cordis.patch.yml — it does not inherit the parent shell. If a custom provider's api_key_env names a key outside the list above (e.g. a self-hosted gateway token), add that variable to the Bundle's env block or it will not reach the bridge.

Development install and removal steps are documented in integrations/deepseek-harness/.

Configuration

Pivot runs on auto-detection by default — supply API keys (UI or shell env) and matching providers are enabled with smart routing defaults. No YAML required for the common case.

For advanced setups, drop YAML into ~/.pivot-web-search/:

File · Purpose providers.yaml · Take over provider config: SearXNG, custom JSON APIs, LLM-search providers, explicit priorities proxies.yaml · Take over proxy config: SOCKS5, forced-proxy (no direct fallback), per-proxy priority

Precedence is all-or-nothing per file: if a file exists, auto-detection for that concern is bypassed — list every entry you want (including DDG). Templates live in examples/.

~/.pivot-web-search/providers.yaml

Providers are tried by priority (lower = first). Same-priority providers are hedged — queried concurrently with staggered starts, first quality result wins. Without an explicit priority, smart defaults apply by type.

providers:
  - name: tavily
    type: tavily
    api_key_env: TAVILY_API_KEY

  - name: brave
    type: brave
    api_key_env: BRAVE_API_KEY

  - name: gemini
    type: gemini
    api_key_env: GEMINI_SEARCH_API_KEY
    model: gemini-2.5-flash

  - name: ddg
    type: ddg

  # Self-hosted SearXNG
  # - name: searxng-local
  #   type: searxng
  #   endpoint: "http://localhost:8888/search"

  # Generic JSON API (Serper, Google CSE, etc.) — multiple instances allowed,
  # each with independent priority, quota tracking, and circuit-breaker state.
  # - name: serper
  #   type: json_api
  #   endpoint: "https://google.serper.dev/search"
  #   api_key_env: SERPER_API_KEY
  #   method: POST
  #   request_body:
  #     q: "{{query}}"
  #     num: "{{max_results}}"
  #   response_mapping:
  #     results_path: "organic"
  #     title: "title"
  #     url: "link"
  #     snippet: "snippet"

Smart default priorities (when no explicit priority):

Type · Priority · Timeout llm_search · 10 · 15s tavily / brave / gemini · 20 · 4s / 4s / 20s searxng / json_api · 30 · 6s ddg · 90 · 6s

Note: an llm_search provider at priority 10 runs ahead of Tavily/Brave with a 15s timeout, so every query may take 15+s. Bump its priority above 20 if latency matters more than answer quality.

LLM Search Providers (type: llm_search)

For any model with built-in web search grounding (Perplexity Sonar Pro, OpenAI with web_search, SAP AI Core, etc.). These return an AI answer plus cited URLs. Configure via ~/.pivot-web-search/providers.yaml (template at examples/providers.yaml).

`chat_completions` — any /chat/completions-compatible endpoint with built-in search:

- name: sonar-pro
    type: llm_search
    api_format: chat_completions
    endpoint: "https://api.perplexity.ai/chat/completions"
    model: sonar-pro
    api_key_env: PERPLEXITY_API_KEY
    timeout: 15

Response parsing tries, in order: search_results (Perplexity/Sonar), annotations with type: url_citation (OpenAI Chat Completions), then top-level citations.

`responses` — OpenAI Responses API (/responses) with the web_search tool:

- name: gpt-web-search
    type: llm_search
    api_format: responses
    endpoint: "https://api.openai.com/v1/responses"
    model: gpt-4o
    api_key_env: OPENAI_API_KEY
    timeout: 45
    search_tool: web_search
    search_context_size: medium

Common fields:

Field · Required · Description api_format · No · chat_completions (default) or responses endpoint · Yes · Full URL to the API endpoint model · Yes · Model identifier api_key_env · Yes · Env var holding the API key (sent as Bearer token) max_tokens · No · Max response tokens (default: 500 / 4000 for responses) timeout · No · Request timeout in seconds (default: 30) system_prompt · No · System prompt (chat_completions only) headers · No · Additional request headers web_search_options · No · Extra search options (chat_completions only)

The gemini type is also LLM-search internally (Google Search grounding) but keeps type: gemini for backward compatibility and dual-key fallback.

~/.pivot-web-search/proxies.yaml

When present it takes over completely — the install-time Proxy URLs field is ignored and direct is not auto-appended (the escape hatch for forced-proxy setups).