ysr666/dsh-vision-router

dsh-vision-router

[!CAUTION]

Data leaves your machine by default. When a cloud vision model is used, Vision Router sends the image (or a derived crop), the vision prompt and related request metadata to that provider. A fresh install has an anonymous OVHcloud vision fallback enabled, so "free" and "no key" do not mean offline. Do not use the cloud chain for confidential, regulated or classified material. See Data flow and strict local-only use.

[!WARNING]

v3.0.5 release notes: fixes Vision reply continuation, optional-peer activation and OVH request-size handling; improves the Vision toggle's transient catalog-refresh behavior. See v3.0.5 notes. For the version actually available to install, check GitHub Releases or npm; release notes are not proof that the version has been published.

Start here

1. Install in an existing DSH Web profile:

npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router

2. Keep your usual chat model and click 👁 Vision beside the composer; check for 👁 Vision ✓.

3. Paste or upload an image and ask the agent what to inspect. The built-in keyless cloud fallback is available out of the box.

Using Desktop? Change --profile web to --profile desktop and reload the app. For Oh-DSH and manual setups, follow the complete installation guide.

Full quick start · What it can do · Data flow and local-only use

Try it on real images

• Understand a screenshot: “What does this error dialog mean? Read the exact message.”
• Find a UI element: “Where is the Save button in this screenshot? Give me its position.”
• Compare two designs: “Compare my page to this reference. What should I fix next?” Use vision_pixel_diff to measure changes after each iteration.
• Read and inspect a long image: “Extract the relevant text, then crop the unclear part to double-check it.”

These are ordinary chat requests: your normal model reasons through the task and calls vision tools as needed. See the tool list or configure your own/local vision backend.

Contents

• Try it on real images
• Highlights
• How it works
• Tools
• Quick start
• Data flow and strict local-only use
• Free vision key channels
• Why this exists
• How it compares
• Configuration
• Install and lifecycle
• Troubleshooting
• Design lineage
• Acknowledgements
• Release notes and community

Highlights

• Capability-aware Auto routing. Keep configured order for deterministic control, or explicitly enable Auto to prioritize already-configured models using measured capability evidence. Auto never infers capability from model names, and enabling Auto alone does not start benchmarks.
• Verifiable model profiling. Exact Test Vision sends one request to one exact model; Quick and Full benchmark OCR / general / structured / document / grounding capabilities. Background profiling is separately authorized and yields to real foreground vision work.
• Original pixels, real answers. The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
• Automatic failover with classified errors. Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
• Image memory. Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
• A verifiable pixel loop. Reference → vision_html_screenshot → vision_pixel_diff (ratio + worst 8×8-grid regions) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
• Stable tool schema. All fourteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. progressiveTools: true remains an advanced boot-time opt-in; only then does vision_activate mount the tools on demand. See docs/progressive-tools-cache.md.
• Selective proxy. Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.

Pixel loop in practice

The agent rebuilt the UI from the reference image, then verified the final result with vision_pixel_diff: 2.54% final diff (32,939 / 1,296,000 differing pixels, threshold 16/channel).

How it works

The vision model is only the eyes; DeepSeek is always the brain. An image turn is never hijacked by a one-shot vision answer — the agent drives the tools itself and can keep operating on the image across as many steps as the task needs.

Tools

Default progressiveTools: false: all fourteen deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set progressiveTools: true in the profile/composition cordis.patch.yml, progressive mode is restored: only vision_activate is exposed initially, the full tool set mounts on first use, and the vision-tools skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:

The diagram covers the eleven image-processing tools. vision_present (durable image delivery) and vision_bootstrap (the optional 1+x structured first pass) bring the default deep-tool set to fourteen. Enabling the privacy-gated vision_screenshot at boot adds an optional fifteenth tool.

Tool · What it does · Artifact vision_bootstrap · Optional 1+x structured first visual pass; establishes task-independent evidence before at least one follow-up vision call · — vision_describe · Image Q&A / multi-image compare / structured-evidence JSON mode (summary + layout regions + entity inventory + verbatim transcription) · — vision_materialize · Copy an authorized attachment into a stable content-addressed workspace handoff file and return absolute + short workspace-relative paths for local OCR/parser fallbacks; no vision/network call · image copy vision_ground · Locate a target → original-pixel box x1/y1/x2/y2 · annotated PNG (optional) vision_detect · Numbered inventory of every element of a kind (buttons/inputs/links…) with original-pixel boxes · annotated PNG with numbered boxes vision_crop · Crop and zoom into a pixel box · PNG vision_present · Publish a generated or edited local image as a durable chat attachment so the user can see it · image attachment vision_pixel_diff · Per-pixel comparison: diff ratio + worst 8×8-grid regions · red heatmap PNG + JSON report vision_colors · Dominant colors (hex + share) · — vision_ocr · Text transcription: configurable default engine (auto / local Tesseract / vision model); an explicit per-call engine still wins · — vision_trace · SVG vectorization (potrace posterization; icons/logos) · SVG vision_extract_foreground · Cutout via border flood fill (uniform backgrounds) · transparent PNG vision_html_screenshot · Screenshot a local HTML file (headless system Chrome); fullPage: true captures the whole page and reports pageHeight · PNG vision_screenshot · Disabled by default; explicit opt-in required. Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses screencapture, and Linux requires ImageMagick import or scrot; optional identify=true tries enabled local recognition backends in order and returns the description with the path · PNG / + description vision_long_screenshot_ocr · Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown · chunk PNGs + Markdown + manifest

Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no .png renaming needed).

Common workflows

vision_ground image="ref.png" target="the send button"
vision_detect image="page.png" target="input fields"
vision_crop   image="ref.png" region="1067,841,1108,881"
vision_present path="rebuilt.png"
vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
vision_ocr image="screenshot.png"
vision_colors image="ref.png" top=8
vision_trace image="icon.png" steps=4
vision_extract_foreground image="logo.png"
vision_html_screenshot source="page.html" width=1200 height=720
vision_html_screenshot source="page.html" width=1200 height=720 fullPage=true
vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120

Quick start

1. Install the plugin

For normal npm/npx installs, installation is a single command:

npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router

[!IMPORTANT]

Which profile? Install into an existing profile that already serves the Web UI:

- Web DSH: --profile web (default data directory ~/.dsh);

- the official DeepSeek Harness desktop app: --profile desktop (same ~/.dsh data directory; reload the plugin or restart the app afterwards);

- Oh-DSH Desktop: its data directory is ~/.ohdsh, see Oh-DSH Desktop below.

dsh plugin add only adds the plugin to the profile you name. A brand-new profile contains just @deepseek-ai/dsh-base, and dsh web never binds a port in that state (no Web surface). Install into an existing profile, or give the new profile a Web surface first.

[!WARNING]

If this profile already loads community plugins manually through cordis.patch.yml, do not mix that legacy setup with dsh plugin add / dsh plugin list: current DSH CLI behavior can also append bundle-patch dependencies to dsh.profile.bundles, causing those plugins to register twice. Migrate the existing manual plugin rows to bundle-managed loading first, or keep using the manual installation path. See deepseek-harness discussion #2889.

[!NOTE]

Third-party dsh-web-plugin-manager / dshpm v0.4.2+ is also compatible: its quality gate now correctly allows @deepseek-ai/schemastery as a runtime dependency. The official DSH CLI above remains the recommended install path.

If you run DeepSeek Harness from a source checkout with pnpm, use the workspace script instead — dsh is not necessarily on your shell PATH:

cd deepseek-harness
pnpm dsh plugin --profile web add dsh-vision-router

If you already installed the DSH CLI globally and dsh is on PATH, the shorter dsh ... form works too. After installation, start or reload DSH Web as you normally do.

[!NOTE]

If you install the plugin into a Web process that was already running long-term, let that DSH Web process reload once so the plugin bundle itself is discovered. After the plugin is loaded, adding/removing models or changing wrapper scope hot-updates without further DSH restarts.

2. Pick your normal model, then enable “👁 Vision” when needed

The stock model selector in the lower-right corner still chooses your brain/conversation model — DeepSeek, Qwen, or any other ordinary route. Vision Router’s generated “+ Auto Vision” wrappers remain real Host routes for image admission, but are hidden from the stock picker and /model when ownership can be established safely.

When you need image input, explicitly click “👁 Vision” beside the composer:

• 👁 Vision: the ordinary model is active and Vision is off;
• 👁 Vision ✓: Vision Router has switched the session to that model’s internal vision wrapper;
• the mode persists across sends and does not auto-reset;
• turning it off switches back to the same ordinary model; choosing a different ordinary model turns Vision off;
• changing only reasoning effort keeps Vision on.

[!IMPORTANT]

Pasting or uploading an image does not enable Vision automatically. Before sending an image, make sure the control shows `👁 Vision ✓`.

The real wrapper route is still present underneath to satisfy DSH image admission. Hiding is presentation-only and fails open: if the browser cannot confidently prove a route belongs to Vision Router, that route remains visible rather than risking hiding a third-party provider.

3. Paste or upload the image

With “👁 Vision” enabled, paste or upload an image normally. By default the complete vision tool schema is stable from session start, so the agent can immediately use vision_describe, vision_ground, vision_crop, and the rest across multiple steps when needed.

If the session already contains images, DSH may reject switching from a vision wrapper back to a text-only route. Vision Router does not bypass that Host rule: it shows a transient error using the same interaction style as the stock model selector, keeps the real current model unchanged, and leaves 👁 Vision ✓ reflecting the actual state so the session stays usable.

The built-in anonymous OVH vision fallback is already configured, so normal image use needs no signup or API key. The lower-right chat picker selects only the brain/conversation model; vision backends do not belong there. Advanced options live under Settings → Vision Router: each vision-backend row may select any callable generative user model already configured under Settings → Models. DSH image-capability metadata is advisory for backend selection, but an explicit text-only declaration makes the Host replace image pixels with text placeholders before adapter dispatch. Vision Router uses a direct pixel bridge only for verified http(s) OpenAI Chat Completions endpoints; otherwise that backend fails safely and the chain continues. WebSocket, RPC, Responses API and other native protocols use their registered adapters when image input is declared or modalities are unknown. If a custom model genuinely supports images, first enable Image under Settings → Models → provider editor → Customized settings → Model options → Input types when that Host control is available. On older Hosts without the control, set the provider-specific model field (llm-pi-ai: input: [text, image]; llm-deepseek: inputModalities: [text, image]) and reload the model configuration. Never claim image input for an endpoint that cannot accept images. Leaving every user row empty is valid; the OVH chain remains the final fallback. Vision HTTP is an internal transport route, not a model group users should select.

See it in action

*Left: an image turn — the user sends a picture, the agent calls vision_describe through the free chain and answers. Right: the finished structured answer.*

Data flow and strict local-only use

Vision Router does not operate fully offline in its default configuration. The exact boundary depends on the operation and the selected models:

Operation · Where data goes Local pixel tools such as crop, pixel diff, palette, SVG trace, cutout, materialize and HTML screenshot · Processed on the DSH machine; these operations do not call a vision model. vision_ocr with Tesseract · Processed on the DSH machine. With the default auto engine, an unavailable or empty local OCR result may fall back to a vision model. Image-turn routing and vision-model tools such as describe, detect, ground or vision OCR · The image or derived crop, prompt and relevant context are sent to the selected vision provider. Built-in free fallback · Sent anonymously (no API key) to OVHcloud AI Endpoints at oai.endpoints.kepler.ai.cloud.ovh.net; the service still receives the request payload and network metadata such as the source IP. User-configured cloud model / HTTP provider · Sent to that provider's configured endpoint under its own retention and privacy terms. Local Ollama / LM Studio · Image pixels are sent to the configured local endpoint. The resulting text still returns to the current chat model, which may itself be remote.

To stop Vision Router itself from sending image data to remote vision endpoints, enable Settings → Vision Router → General → Local-only vision. This is a runtime policy, not a destructive rewrite: saved cloud rows and freeFallback stay configured but cannot execute while the switch is on. Only loopback visual endpoints (localhost, 127.0.0.0/8, ::1) are eligible, including Ollama, LM Studio, and custom local HTTP backends. For a strict local-only workflow, also use a local chat model because vision result text still returns to the current chat model. Test the final network boundary in your own environment; Vision Router cannot make a remote chat model, proxy or Host integration local.

Free vision key channels

The built-in OVH fallback is anonymous by design, and OVH caps anonymous use at 2 requests/minute per IP per model. If that feels tight, the channels below list other vision-model options and free or trial offers. Free quotas, authentication, third-party client access and image-input support vary; a model listed in a provider's catalog is not a guarantee it can be called from DVR. This is an October 2026 reference, not a live quota guarantee; verify your specific API route and model before relying on it.

Channel · Free vision model(s) · Free quota · CN direct? · Where to get the key OVHcloud AI Endpoints (access key) · Qwen2.5-VL-72B-Instruct — the same endpoint the built-in fallback uses · 400 req/min per project per model (vs 2 anonymous) · ✅ · OVH account → Public Cloud project (attach a payment method; free models are not charged) → AI Endpoints access key Zhipu (bigmodel.cn) · glm-4.6v-flash · glm-4.1v-thinking-flash · glm-4v-flash — three permanently free models; chaining them triples capacity · uncapped tokens · ✅ · open.bigmodel.cn → API keys DashScope (Aliyun) · qwen3-vl-flash (limited-time free) and the Qwen-VL series · new users: 1M tokens per model series / 90 days · ✅ · bailian.console.aliyun.com Intern AI (Shanghai AI Lab) · internvl-latest · internvl3.5-latest · 30 RPM, 90M tokens/month · ✅ · chat.intern-ai.org.cn Groq · meta-llama/llama-4-scout-17b-16e-instruct (native multimodal, up to 5 images) · 30 RPM / 14,400 req/day, no card · ❌ proxy · console.groq.com Google AI Studio · gemini-2.5-flash · gemini-2.5-flash-lite · 10–30 RPM / 500–1,500 req/day · ❌ proxy · aistudio.google.com NVIDIA NIM · meta/llama-3.2-11b-vision-instruct · nvidia/nemotron-nano-12b-v2-vl · 40 RPM, no card · ⚠️ · build.nvidia.com OpenCode Zen · mimo-v2.6-flash-free (listed in Zen; third-party image requests not independently verified) · Limited-time free in OpenCode; no verified fixed external-API quota · ⚠️ · OpenCode Zen OpenRouter · google/gemma-4-26b-a4b-it:free · google/gemma-4-31b-it:free · 50 req/day on unpaid accounts · ❌ proxy · openrouter.ai

Channels that permit third-party API access and accept image content can join the vision chain as an httpProviders entry (key in the matching environment variable or ~/.dsh/.credentials.yaml); the chain tries those entries before the anonymous fallback. OpenCode Zen caveat: its docs currently list both mimo-v2.5-free and mimo-v2.6-flash-free. Xiaomi's October 21 deprecation concerns the direct Xiaomi model IDs, not a confirmed Zen alias shutdown. Free Zen access may be restricted to OpenCode clients; do not assume the model works in DVR or try to bypass such restrictions. Zen warns that usage of its free MiMo model may be collected to improve the model. OpenCode Zen documentation · Xiaomi deprecation notice.

[!NOTE]

Free-tier policies change without notice — Cerebras retired its free tier in July 2026 (now a one-time $5 credit), SambaNova's free tier is down to 20 requests/day, and Hugging Face's is $0.10/month. Third-party “:free relay” aggregators are deliberately not listed: they rotate quickly, lack SLAs, and some resell quota in ways that violate upstream terms.

Why this exists

Most DSH vision plugins bridge ima