dsh-tool-describe-image

dsh-tool-describe-image — Image Understanding Tool Plugin

English | 中文

Model-facing describe_image tool: gives text-only models (DeepSeek V4 etc.) image understanding. Each call loads one image — a local file path, an http(s) URL, or a session attachment reference — and asks an OpenAI-compatible vision endpoint (Qwen-VL, GLM-4V, GPT-4o, a local Ollama endpoint…) to answer over the Chat Completions or Responses protocol; only the returned text enters the conversation, the image itself never enters the session log.

Ported from deepseek-harness packages/vision/tool-describe-image (mirrored at whitelonng/dsh-plugin-describe-image), adapted to the dsh-web-ui family conventions: official NPM SDK only, host-side plugin with a browser half, live settings, no dsh source changes.

Capabilities

Security model

• Vision requests and image downloads both refuse HTTP redirects (redirect: 'error'); bearer credentials and image bytes never reach a source other than the configured deployment.
• The request body carries the base64 image but no key; request headers and resolved credentials are not logged.
• Only http(s) URLs and local paths are accepted; every other URL scheme is rejected.
• The attach route validates base64, magic bytes, and the byte bound before the attachment store persists anything; only the reference JSON (text) crosses into the conversation.
• Response bodies are truncated at the cap (maxOutputTokens * 8 + 64 KiB) before parsing.

Installation

Install the family aggregate @linxin666/dsh-web-ui-all (all plugins and skins in one package), or this plugin alone:

# Recommended: install directly from npm
dsh plugin --profile web add @linxin666/dsh-tool-describe-image

The aggregate mounts this plugin without configuration: loading is unaffected, and the first call fails with a clear error (describe-image: baseURL must be an absolute http(s) URL) until configured. Fill in the endpoint and model on the "Image understanding" card under Settings → Plugin config to start immediately, no restart needed. (Difference from upstream: upstream validates eagerly at load; the family aggregate has no config entry, so validation is eager only when a composition entry actually configures it and per-call otherwise.)

Configuration

Configured mount example (profile cordis.patch.yml / composition file):

- id: describe-image
  name: '@linxin666/dsh-tool-describe-image'
  config:
    baseURL: https://dashscope.aliyuncs.com/compatible-mode/v1
    model: qwen-vl-max
    apiKey: !!js process.env.VISION_API_KEY

Endpoints exposing only the Responses API set apiStyle: responses:

- id: describe-image
  name: '@linxin666/dsh-tool-describe-image'
  config:
    baseURL: https://api.openai.com/v1
    apiStyle: responses
    model: gpt-4o-mini
    apiKey: !!js process.env.VISION_API_KEY

Usage

Custom instructions

The tool takes a prompt argument: tell the vision model exactly what you need — "transcribe all text", "extract the table as CSV", "diagnose the UI layout problems", "translate the text into Chinese". A targeted instruction beats a generic description; the tool description steers the text model toward passing one. Calls without a prompt fall back to defaultPrompt.

Sending images directly

Text-only models have no image entry in the DSH input box, so drag or paste an image into the composer: at send time the plugin rewrites the image-bearing send into a describe-image reference (![图片](/describe-image/raw/sha256:…)) instead of an image block the model cannot read — the image renders in the conversation and the model analyzes it through the tool. The bytes travel to the host /describe-image/attach route (validated for size and magic bytes, persisted in the attachment store); only the reference text enters the session log.

Known limitations

• Only the magic-byte gate checks the type; the image is not decoded, so a header-valid but corrupt file fails only at the vision endpoint.
• One image per answer: no multi-image input, no follow-up on the previous image, no structured output (coordinates / boxes).
• Extracting text still costs one VLM call: OCR-only deployments can point baseURL at a cheaper OCR model.
• OpenAI-compatible protocol only: Chat Completions (/chat/completions) and Responses (/responses) are supported; vendors with other request/response shapes need separate adapters.

Source and copyright

• Source: ported from whitelonng/dsh-plugin-describe-image (deepseek-harness packages/vision/tool-describe-image), moved in 2026-08; tests ported with the source (pnpm --filter @linxin666/dsh-tool-describe-image test).
• Copyright: the original code belongs to its authors (deepseek-ai / whitelonng); this repository only hosts and maintains it and claims no copyright; the ported contribution is licensed by its contributor under the family license.
• License: the family is licensed under Apache-2.0 (repository root LICENSE); this package's license field is Apache-2.0.