dsh-vision-pro-bridge

dsh-vision-pro-bridge

Give a text-only DeepSeek-V4-Pro brain real image understanding — without switching models, without Ollama, and without a second API key.

DeepSeek Harness (dsh) plugin. English · 简体中文

The problem

deepseek-v4-pro (and deepseek-v4-flash) are text-only models: their API rejects image bytes. In DeepSeek Harness, pasting a screenshot into a Pro session fails with "does not accept image input". Your only official option is to switch the whole conversation to the weaker, Flash-tier deepseek-v4-flash-vision-exp.

What this plugin does

It registers a twin provider route — deepseek-vision-pro (model deepseek-v4-pro-vision) — that advertises image support. When you attach an image, the plugin first asks deepseek-v4-flash-vision-exp to transcribe it into text (using your existing DEEPSEEK_API_KEY), then hands that text — plus a stable local file path — to deepseek-v4-pro. The DeepSeek wire route never sees an image; V4-Pro keeps its coding brain and still "sees" the picture.

paste screenshot  →  [image block admitted]
   → transcribe via deepseek-v4-flash-vision-exp (text)
   → text + local path  →  deepseek-v4-pro answers

Why this one

• Zero third-party dependencies. It reuses only the harness's own @deepseek-ai/dsh-llm and @deepseek-ai/dsh-llm-deepseek. No openai, no sharp, no schemastery, no Ollama.
• DeepSeek-only, single provider. Images go only to DeepSeek (deepseek-v4-flash-vision-exp), through the same DEEPSEEK_API_KEY. No second VLM vendor, no local model, no anonymous endpoints.
• Official image pipeline, byte-for-byte. It subclasses the official DeepSeekAdapter, so image normalization, Files-API upload, and inline-base64 fallback are identical to official behavior — no hand-rolled HTTP.
• Minimal & auditable. One file, ~250 lines. Easy to read, verify, and fork.
• Content-addressed persistence. Images are saved under ~/.dsh/vision-pro-bridge/images/ and the stable local path is passed to the model, so V4-Pro can reference it.
• Pro-focused. One model, deepseek-v4-pro-vision: keep V4-Pro's coding ability, add vision.

Install

dsh plugin --profile web add dsh-vision-pro-bridge
# restart dsh web

Usage

1. In the model picker, choose DeepSeek-V4-Pro (视觉桥).
2. Paste or drag a screenshot and ask normally.

Make it the default for new sessions (~/.dsh/settings.yaml):

agent-default-model:
  provider: deepseek-vision-pro
  model: deepseek-v4-pro-vision

Configuration

Environment variables (all optional):

Images are cached content-addressed under ~/.dsh/vision-pro-bridge/images/ (re-attaching the same image reuses the transcription); an activity log is written to ~/.dsh/vision-pro-bridge-activity.jsonl.

How it works

1. apply(ctx) registers a DeepSeekAdapter subclass for the provider route deepseek-vision-pro.
2. listModels / resolveModel / prepareCall all declare inputModalities: ['text', 'image'], so the harness's image gates (paste admission, thumbnails, read_image) open. Overriding prepareCall is required — otherwise the LLM runtime projects images to placeholders before they ever reach the adapter.
3. stream() walks every message (including nested tool-result content), and for each image block reads its bytes, saves them content-addressed, and calls super.stream() against deepseek-v4-flash-vision-exp to produce a text transcription.
4. The image block is replaced with [图片（视觉桥分析 by …）]\n<transcription>\n\n本地路径：<path> and the now text-only conversation is streamed to deepseek-v4-pro.

With no image, stream() forwards directly to Pro — zero overhead.

Limitations

• Transcription is lossy: great for layout / OCR / UI review, not for exact pixel coordinates.
• The transcription model is fixed to deepseek-v4-flash-vision-exp.
• The endpoint is resolved from DEEPSEEK_BASE_URL (not the settings-page llm-deepseek overrides).
• No fallback chain: if the transcription call fails, the image becomes a placeholder and the turn continues.

Alternatives

For a feature-rich take (fallback VLM chain, local Ollama, settings-page UI, image downscaling, classified errors), see dsh-vision-proxy. This plugin is the minimal, DeepSeek-only, zero-dependency version of the same idea.

License

MIT