dsh-plugin-auto-vision

dsh-plugin-auto-vision

DeepSeek Harness (DSH) Smart Multimodal Vision Enhancer Plugin

Resolves image recognition degradation and the "cannot see/recognize images" issue for third-party Claude, GPT-4o, Gemini, and open-source VL models.

🌟 Problem Statement

In DeepSeek Harness (DSH), when third-party multimodal models (e.g. claude-3-5-sonnet, gpt-4o, gemini-1.5-pro, qwen2-vl, etc.) are configured via custom providers (OneAPI, NewAPI, CLIProxyAPI, OpenRouter, ModelScope, reverse proxies), users frequently encounter this issue:

When uploading an image, the model replies: "I cannot view images / I cannot recognize images / As an AI I only accept text."

*(Ref: GitHub Discussion #112)*

🔍 Root Cause in DSH Core

1. Fallback to Text-Only: Unless a model explicitly defines input: [text, image], the pi-ai adapter defaults to DEFAULT_INPUT = ["text"].
2. Silent Image Stripping: DSH's projectImagesForTextModel() intercepts requests when a model lacks the image modality and silently replaces image blocks with:

[image omitted because this model accepts text only; attachment sha256:xxxx]

The remote model literally never receives the image binary data!

1. Manual Config Pain: Hand-crafting YAML overrides is tedious and easily overwritten by Web UI saves.

✨ Features

• 🚀 Zero-Configuration: Works out of the box after installation. No manual YAML editing needed.
• 🧠 Heuristic Vision Rules: Built-in regex patterns covering Claude 3/3.5/3.7/4, GPT-4o/4.5, Gemini, Qwen-VL, GLM-4V, InternVL, Llama 3.2 Vision, and more.
• 🛡️ Runtime Interception: Hooks into ctx.llm and adapter pipelines to inject image input modality dynamically, preventing image stripping.
• 🩺 Diagnostic Tool: Comes with auto_vision_diagnose for instant channel and model capability scans.
• 🎨 Web UI Badge: Displays a Vision Active badge inside the message composer.

📦 Installation

Run inside the DSH terminal or system terminal:

dsh plugin install dsh-plugin-auto-vision

Or ask the agent in any active conversation:

*"Install the dsh-plugin-auto-vision plugin for me."*

🛠️ Diagnostics

Ask the agent:

*"Run auto_vision_diagnose to check my configured models."*

Returns the modality health status of all registered providers and models.

⚙️ Configuration Schema

Configurable via Web UI Settings → Plugins → auto-vision or cordis.patch.yml:

Field · Type · Default · Description enableAutoInjection · boolean · true · Whether to automatically inject image input modality at runtime customVisionPatterns · string[] · [] · Custom regex strings for matching vision model IDs excludeModels · string[] · [] · Blacklist of model IDs to exclude from image injection

📄 License

Licensed under the MIT License.