dsh-plugin-vision

dsh-plugin-vision

English · 简体中文

Give a DSH agent a pair of eyes.

When the main model is text-only, an image in the conversation is just a line saying [image omitted because this model accepts text only; attachment sha256:1a2b3c4d]. This plugin closes that gap: the agent writes its own observation prompt, hands the image to an online vision model (VLM), and gets a written answer back.

• Many providers — every call is the OpenAI-compatible POST {baseUrl}/chat/completions. Eleven presets ship in the box, and any other OpenAI-compatible endpoint works too.
• Checked capabilities become prompt text — tick what this model is good at (grounding / OCR / documents & charts / video / GUI / deep thinking / multi-image …) and those notes are injected into the agent's system prompt in real time, telling it what else this model can do and how to ask.
• The prompt belongs to the agent — the plugin ships no "describe this image" template. What to look at, how finely, and in what output shape is written by the agent on the spot.
• It can see images from the conversation — whatever the user drops into the input box, the agent can name with latest or with the eight-hex digest it sees in the placeholder.
• Keys live in the credential store — written to ~/.dsh/.credentials.yaml; the UI only ever shows configured or not set, never the literal.
• Bilingual UI — the settings card follows DSH's own language preference (中文 / English).

---

Install

A. Official dsh plugin add (recommended)

⚠️ Do not run dsh plugin add dsh-plugin-vision. That bare name resolves to the public npm registry, where dsh-plugin-vision belongs to an unrelated project by a different author — it is not this plugin and will not work with a current DSH. Always install from this repository.

dsh plugin --profile web add github:aijunjiang/dsh-plugin-vision
# or from a local checkout: dsh plugin --profile web add <this-repo-path>

The internal @deepseek-ai packages ride the DSH baseline as optional peers, so npm pulls nothing; the package declaring dsh.bundle is auto-registered into the profile's bundles layer. Uninstall: dsh plugin --profile web remove dsh-plugin-vision.

Restart dsh web once after installing so the settings card appears.

B. Local development

Link the repository into the target profile's node_modules under its package name, then add one row to that profile's cordis.patch.yml:

# Windows, using the web profile
New-Item -ItemType Junction `
  -Path  "$env:USERPROFILE\.dsh\profiles\web\node_modules\dsh-plugin-vision" `
  -Target "C:\path\to\dsh-plugin-vision"

# ~/.dsh/profiles/web/cordis.patch.yml
- insert:
    - id: vision-bundle
      name: dsh-plugin-vision
      config:
        provider: ark
        apiKeyEnv: ARK_API_KEY

The web profile is live-reloading: saving the patch file applies configuration changes (provider, model, base URL …) immediately. But changes to the plugin's own source (index.js, catalog.js) are not hot-reloaded — the module stays in the ESM cache, so restart dsh web after editing it. The settings card is the browser half: refresh the page once; if it still does not appear, restart dsh web.

⚠️ On Windows, never edit that YAML with PowerShell 5.1's Get-Content / Set-Content -Encoding utf8: the first decodes as GBK and destroys non-ASCII comments, the second writes a BOM. Either one makes the whole patch layer fail to parse and takes every plugin down with it. Use [System.IO.File]::ReadAllText/WriteAllText($p, $t, (New-Object System.Text.UTF8Encoding($false))).

---

Configure

Settings → Plugins → Vision.

Key resolution order: the apiKey settings field → the DSH credential domain (whatever apiKeyEnv points at) → an environment variable of that name.

Bundled provider presets

A preset is a starting point, not a cage: every endpoint and model can be overwritten in the card. Model IDs move fast — the vendor's own documentation wins.

---

How the agent uses it

vision_analyze

Each entry of images may be:

vision_list_images

No parameters. Lists every image attachment seen in this conversation (name, size, type, digest), plus whatever dev machines have pushed into the ingest inbox (newest first, with push time and expiry), so an agent staring at [image omitted …] still knows which images exist and how to name one.

⚠️ Inbox images are not session events — latest will not reach them. Name them by sha256 digest.

---

Pushing images from a dev machine

Dev machines are many, variously configured, and constantly changing. Copying each image onto the host before analysing it meant redoing the same dance for every new machine. Now the host mounts one HTTP route: any dev machine pushes the bytes with a single curl — nothing to install, no port to open.

One push

# Same address you open the GUI on; the token lives in Settings → Plugins → Vision
curl -T shot.png \
  -H "Authorization: Bearer <token>" \
  "http://<host>:3080/dsh-vision/ingest?name=shot.png"

{
  "ok": true,
  "id": "sha256:c414cd0e204d…",
  "digest": "c414cd0e204d",
  "mediaType": "image/png",
  "width": 640,
  "height": 480,
  "expiresAt": "2026-10-23T05:23:23.773Z",
  "usage": "vision_analyze images=[\"sha256:c414cd0e204d\"]"
}

usage is a call you can hand straight to the agent. The bytes never enter the agent's context — only that one digest line does.

Endpoint contract

Where the token comes from

Generated on first boot into ~/.dsh/vision-inbox/token (mode 0600) and reused across restarts, so a dev machine hardcodes it once.

Why the service lives on the host, not the dev machine

Because the thing that needs to see the image is the VLM the host talks to (baseUrl points at a public DashScope / Ark endpoint). Put the service on a dev machine and the host still has to pull it. Put it on the host and each dev machine pushes once. However many dev machines there are, only "who pushes" varies — "how it is received" stays in one place.

Storage and expiry

• Bytes go through the existing attachment service (attachments.saveImage): content-addressed and deduplicated — pushing the same image twice does not cost a second inbox slot.
• Metadata lives in ~/.dsh/vision-inbox/items/ and expires after 24 hours by default (swept every 10 minutes).
• Expiry removes metadata only, never the attachment object — a conversation may still reference it.

Related settings

⚠️ Port 3080 binds 0.0.0.0 with no platform-level authentication. Token checking, the byte cap and the item cap are all implemented inside this plugin. Do not paste the token anywhere public; if in doubt, switch the endpoint off.

How images should reach the model (in the system prompt, and enforced at runtime)

Exception: when baseUrl itself points at a local model (Ollama / vLLM), the model and the image share one network, so private addresses are legitimate and are not blocked.

JSON image bundles: the right way when all you have is base64

Sometimes there really is no file — the bytes came back from a remote API, a database column, a script. Then do not let the agent print the base64. Write it to a JSON file on the side that produced it and let the agent pass a single path:

{
  "images": [
    { "name": "first-frame", "data": "iVBORw0KGgoAAAANSUhEUg..." },
    { "name": "last-frame",  "data": "iVBORw0KGgoAAAANSUhEUg..." }
  ]
}

images: ["json:/tmp/bundle.json"]        the whole bundle
images: ["json:/tmp/bundle.json#2"]      only the 2nd image
images: ["json:/tmp/bundle.json#2-4"]    images 2 through 4

What it saves — measured on three 200×150 PNGs:

The model receives exactly the same images; what disappears is the copy in the agent's context.

The parser is deliberately liberal (Postel's law) — all of these are accepted:

One line on the producing side:

# Remote box: bundle every PNG in the current directory
python3 -c "
import base64, glob, json
print(json.dumps({'images':[{'name':p,'data':base64.b64encode(open(p,'rb').read()).decode()} for p in sorted(glob.glob('*.png'))]}))
" > bundle.json

// Node: dump in-memory buffers straight into a bundle
fs.writeFileSync('bundle.json', JSON.stringify({
  images: buffers.map((b, i) => ({ name: `frame-${i}`, data: b.toString('base64') })),
}))

Limits: each image still obeys the per-image byte cap, the expanded total still obeys max images per call, the whole file is capped at 128 MB, and a bundle may not reference another bundle.

Coordinate discipline

For anything positional, pin the output shape in the prompt and make the model report the pixel size of the image it actually processed — otherwise the coordinates cannot be mapped back:

Output JSON only: [{"label":"...","bbox_2d":[x1,y1,x2,y2]}]
Before the JSON, state on one line the pixel width and height of the image you processed,
and whether the coordinates are absolute pixels or normalized.

Vendors disagree here (known: Qwen-VL returns bbox_2d in absolute pixels of the processed image; GLM-V returns 0–1000 normalized values wrapped in <|begin_of_box|> … <|end_of_box|>), so asking the model to state its own basis beats assuming one.

---

Troubleshooting

---

Development

npm run sync    # regenerate the CATALOG mirror inside client.js from catalog.js
npm test        # host logic + execute path + offline card rendering (no browser)
npm run check   # syntax check + mirror-is-in-sync check + the full test suite

• catalog.js — providers and capabilities, the single source of truth. Adding a provider means editing this file and running npm run sync.
• index.js — the host half: two tools, the settings namespace, key resolution, image resolution, the live system-prompt section, and the mount point for the ingest endpoint.
• ingest.js — the LAN ingest endpoint: token, byte and item caps, content-addressed storage, TTL sweep. Self-contained; the only DSH dependency (attachments) is read optionally.
• client.js — the browser half: the settings card and its zh / en dictionaries. A client bundle may not import, so the catalog is mirrored in by generation rather than imported.
• tests/session-images.test.mjs — conversation image enumeration (the ground latest and digests stand on): all five event shapes the host's session controller walks (data.content, data.message.content, data.inserted[], the block-end of assistant/chunk, and nested tool-result), plus de-duplication, ordering, hostile data, config normalization and prompt injection.
• tests/execute.test.mjs — stands up a local OpenAI-compatible endpoint and drives execute() end to end: request shape (auth header, per-vendor detail/thinking mapping, multi-image labelling, extraBody merge) and every response/error branch (chunked content, truncation, thinking-only, refusal, HTTP 500, non-JSON, missing key, too many images, abort).
• tests/client-card.test.cjs — renders the card to static HTML with react-dom/server and asserts the official card structure, both languages, zh/en key parity, the collapsed advanced section, and the save/credential call sequence.
• tests/ingest.test.mjs — the ingest endpoint: stands up a real http server and drives it with real requests, covering token check (missing and wrong answer identically), methods, formats, the byte cap, the item cap, content-addressed dedup, TTL sweeping, corrupt-metadata tolerance, and token persistence.

ingest.test.mjs needs only node:*, so it always runs; the other three skip cleanly (exit 0) when their dependencies are absent: index.js needs @deepseek-ai/schemastery, and the card test needs react / react-dom (injected by the DSH platform in the browser; locally they resolve from the DSH profile's node_modules).

Working without a real key

Stand up an OpenAI-compatible fake endpoint, point baseUrl at it and put anything in apiKey:

// mock.cjs — dump whatever arrives, then answer with a fixed line
require('node:http').createServer((req, res) => {
  const chunks = []
  req.on('data', c => chunks.push(c))
  req.on('end', () => {
    require('node:fs').writeFileSync('last-request.json', Buffer.concat(chunks).toString())
    res.writeHead(200, { 'Content-Type': 'application/json' })
    res.end(JSON.stringify({ choices: [{ message: { content: 'MOCK OK' }, finish_reason: 'stop' }] }))
  })
}).listen(8791, '127.0.0.1')

License

MIT