tool-vision

dsh-tool-vision

GitHub: Scorp1o117/dsh-tool-vision · npm: dsh-tool-vision

Part of the DeepSeek Harness Enhancement Suite — Vision · Soul/Persona · Long-term Memory · Plugin Marketplace.

External vision model for DeepSeek Harness.

DeepSeek's own models are text-only, and the harness derives every model request strictly from the session log (llm/stream requests must equal the durable derivation — the agent-loop invariant). This plugin bridges the gap in two ways:

1. inspect_image tool — sends an image (local file, or http(s) URL) to any OpenAI-compatible /chat/completions endpoint that supports image_url content parts, and returns the vision model's textual answer into the agent loop.
2. Image bridge (v0.2.1) — pasted images are turned into inspect_image hints before they enter the durable log, on the agent/pre-step waterfall (the one seam where the harness lets a plugin replace the messages of a proposed step). Images already logged by an older version are repaired lazily with a surface replace on the session's first pre-step. Only models listed in multimodalModels receive image blocks directly; a model's declared inputModalities are never consulted, because profiles routinely declare input: [text, image] on text-only models just to pass the harness's prompt-admission check.

• Zero dependencies beyond the dsh SDK — works with any compatible endpoint: OpenAI GPT-4o, Qwen-VL (DashScope), GLM-4V (Zhipu), Moonshot, Gemini compatible endpoints, local Ollama, etc.
• Registered on the global tools layer: every agent in the process can call inspect_image.
• Web UI settings section (v0.3.0): Settings → 视觉模型 edits the tool-vision namespace (API endpoint, write-only key, model, bridge options) in settings.yaml; changes hot-apply without a restart. The API key lives in settings.yaml, not the profile patch. Mount by package name (name: 'dsh-tool-vision') so the web client bundle is discovered.

Install

Mount in a profile patch ($DSH_HOME/profiles/<name>/cordis.patch.yml):

- insert:
    - id: tool-vision
      name: 'dsh-tool-vision'     # after: pnpm add dsh-tool-vision in the profile
      config:
        baseURL: 'https://api.openai.com/v1'
        apiKeyEnv: 'VISION_API_KEY'
        model: 'gpt-4o-mini'

Or load it from a local path without npm:

    - id: tool-vision
      name: './plugins/dsh-tool-vision/index.js'

Config

Image bridge setup

1. In your model settings, declare image input on the models you paste images onto, so the harness admits image messages (pi-ai style):

   llm-pi-ai:
     providers:
       your-provider:
         models:
           - id: deepseek-v4-flash
             input: [text, image]
   

2. List genuinely multimodal models in the plugin config so they receive image blocks untouched:

   - id: tool-vision
     name: 'dsh-tool-vision'
     config:
       multimodalModels: ['mimo-v2.5', 'grok-4.5']
   

Then pasting an image while on a text-only model stores a hint like [User sent an image, exported to: <path>. Inspect it with the inspect_image tool...] in the transcript (the pasted image no longer renders as pixels in that message), and the agent inspects it through the configured vision endpoint.

Why not llm/stream? The harness freezes every request and the agent-loop invariant fails any request whose messages diverge from the session-log derivation (log-reconstruction desync), and this cordis waterfall's next() cannot replace request arguments. The agent/pre-step waterfall is the supported seam: its decision messages become the durable log, so the invariant stays satisfied.

Key resolution order: config.apiKey → process.env[apiKeyEnv] → process.env.OPENAI_API_KEY.

Tool: inspect_image

Example endpoints (baseURL):

• OpenAI: https://api.openai.com/v1 — gpt-4o, gpt-4o-mini
• Alibaba DashScope (Qwen-VL): https://dashscope.aliyuncs.com/compatible-mode/v1 — qwen-vl-plus, qwen-vl-max
• Zhipu (GLM-4V): https://open.bigmodel.cn/api/paas/v4 — glm-4v-flash (free tier), glm-4v-plus
• Moonshot (Kimi): https://api.moonshot.cn/v1 — moonshot-v1-8k-vision-preview
• Ollama local: http://localhost:11434/v1 — llama3.2-vision (no key)

Note for users

• This plugin is a standard profile bundle (dsh.bundle.patch): dsh plugin --profile web add dsh-tool-vision installs and mounts it in one step — no manual cordis.patch.yml edits needed.
• The settings section needs the dsh-host-apiproxy namespace allowlist; the plugin patches it automatically on first start — restart dsh web once more and the section appears. A dsh update overwrites the patch; the next plugin start re-applies it.
• Settings changes hot-apply (no restart needed).
• Tested against DSH 0.1.0-rc.6.

Pixel-level vision tools (v0.4.0, ported from dsh-vision-router)

14 vision_* tools driven by the same configured endpoint as inspect_image (baseURL/apiKey/model) — no provider chain, no local models, no extra settings:

Quality & safety details:

• Content-hash cache keyed by endpoint+model+image+question (no stale answers across model switches, failures are never cached).
• Uniform 4MP downscale before every model call; oversized inputs are rejected with a clear error (stat pre-check, 20MB cap on both file and attachment paths).
• Rate-limit / 5xx auto-retry with Retry-After-aware backoff; endpoint content-safety rejections are surfaced as VISION_CONTENT_FILTERED instead of a generic backend error.
• Long-OCR bounds: 120s total budget, 40-chunk cap, cancellation checks, stop-on-first-backend-failure.
• Path containment for relative inputs; artifacts land in <workspace>/.dsh-tool-vision/.
• Bridge marker: pasted images become a short [图片: <path>] marker and the usage rule lives in a system-prompt section (clean transcript, same model behavior).

Requires sharp / potrace / puppeteer-core (declared in dependencies; missing ones degrade lazily with an install hint and never break other tools).

Limitations

• A bridged image enters the conversation as a text hint (a transcript, not pixels) — pixel-precise in-context reasoning is not available to text-only models; the vision model's description comes back through inspect_image.
• Images are base64-transferred; mind privacy and size limits.
• Independent of the dsh-llm routing/retry system; failures return clear errors to the agent.

License

MIT