AtropinolTT/dsh-guide-dog

Guide Dog for DSH, powered by MiniMax

English | 简体中文

A dynamic Cordis plugin that gives DeepSeek Harness multimodal superpowers through the mmx CLI (MiniMax):

• Eyes for DeepSeek — MiniMax VLM (guide_dog_vision / guide_dog_inspect) describes images, so a model with no native vision input (e.g. DeepSeek) can still review frontend designs, figures, screenshots, and generated images.
• Hands for generation — images (image-01), video (MiniMax-H3 / Hailuo), speech (MiniMax TTS), music (music-3.0), text (MiniMax-M3), and web search.
• Web UI preview & playback — every generated file is served same-origin at /guide-dog/media/<file> and rendered inline in the conversation tool cards (<img>, <audio controls>, <video controls>), plus a Guide Dog settings page with auth status and a speak tester.
• Skill integration — guide_dog_speak reuses your existing audio-conversation and speech-mmx skill pipelines (text transform, CJK auto-detect, per-language voices, host playback), and falls back to raw mmx speech synthesize when the skill scripts are absent.
• Automatic invocation — a mounted system-prompt section (guide-dog-vision, order 110) tells the agent to auto-invoke the inspection tools for any job needing visual checks, especially when the active model cannot see images.
• Call mode (Phase 2, shipped) — hands-free, real-time voice conversation in the web UI: VAD / push-to-talk turn-taking, streaming sentence-level TTS with barge-in, consensus-first protection for write commands, progress announcements, and a unified floating dual-pill UI at the composer with zh/en i18n (details in "Phase 2 — call mode" below).
• Accessibility mode (Phase 3, planned) — an a11y config block is reserved (auto-narration, vision-cloud, summary-first); accessibility features are next on the roadmap and will be tested and rolled out after the call-mode shakeout.

Featured

Guide Dog is featured in dsh-recommend, a community-curated plugin directory for DSH. It passed the project's certification review and carries the gold certified badge at the top of this README; the score badge updates automatically on every registry sync.

Files

Install

Install straight from this repository with the standard DSH plugin command — the root package.json declares the dsh.bundle manifest, so the profile's pnpm layer resolves the package itself (no npm publish required):

dsh plugin --profile web add github:AtropinolTT/dsh-guide-dog

Deploy (static web-profile bundle — current)

1. Edit the source of record: plugin-host.js / plugin-client.js.
2. python3 deploy/convert_bundle.py — regenerate bundle/lib/.
3. python3 deploy/publish.py — copy to ~/.dsh/dsh-guide-dog, idempotently register in ~/.dsh/profiles/web (dependency link + bundles entry + node_modules symlink), remove the superseded autoload bundle.
4. Restart DSH (dsh web) — bundles are parsed at startup.

No dynamic plugin, no approval cards, no per-session instances: after a DSH restart the tools and voice UI come back with the profile itself. Full details and pitfalls in the "Restart recovery" section below.

plugin-source.js is a legacy dynamic-era artifact (both halves concatenated); kept for reference, not used by the current deploy flow.

Tools

* required

Auto-invoke contract (visual checks)

While the plugin runs, a system-prompt section instructs the agent:

• For visual checks (frontend design review, figure/plot/chart generation, screenshots, UI mockups, generated-image QA) it MUST call guide_dog_inspect (structured) or guide_dog_vision (general) on the produced image file before finalizing — never claim to have seen an image it has not inspected.
• Generated media is served to the user at /guide-dog/media/<file>; the agent must include the returned url fields so the user can preview.
• Speech requests route to guide_dog_speak.

Example visual-check flow on DeepSeek:

1. (agent) create figure/screenshot file, e.g. chart.png
2. (agent) guide_dog_inspect { image: "chart.png", focus: "figure" }
          → structured review of axes/labels/readability/encoding
3. (agent) iterate the figure, re-inspect, then finalize with the url
4. (user)   previews chart.png in the web UI card

Media store & serving

• Media lives in ~/.dsh/guide-dog/.guide-dog/media — the global store under GLOBAL_ROOT = ~/.dsh/guide-dog (one instance for the whole web profile since 2026-08-16; no longer the per-workspace sandbox root — see "Restart recovery" below).
• Served by a same-origin prefix route /guide-dog/media with:
  • extension allowlist (jpg/jpeg/png/gif/webp/mp3/wav/m4a/ogg/mp4/webm),
  • basename-only lookup + traversal guard,
  • Accept-Ranges: bytes with real byte-range responses (video seeking),
  • 404/405/413/416 as appropriate.
• .index.json keeps metadata (prompt, voice, ts, kind) for the settings gallery (guide-dog/list-media RPC). A corrupt index is rebuilt from the directory.
• Files persist across plugin restarts; stopping/removing the plugin only removes the runtime registrations, never the files.

Skill integration (audio-conversation / speech-mmx)

guide_dog_speak honors the exact pipeline of your two skills:

1. ~/.agents/skills/audio-conversation/scripts/transform.py — markdown/code/URL stripping (falls back to a built-in JS transform when absent).
2. CJK auto-detect → per-language voice defaults (English_Trustworthy_Man / Chinese (Mandarin)_Gentle_Youth), same as the skill env contract. Explicit voice overrides; language boosts accents.
3. ~/.agents/skills/speech-mmx/scripts/mmx_tts.py speak --input … --out … (falls back to mmx speech synthesize).
4. Browser playback via the returned mp3 URL. With playOnHost: true the host speakers play it too — one file at a time (previous playback is terminated first), mirroring the skill's latest-only rule.

Env vars of the skills that still apply when set in the dsh process environment: AUDIO_CONVERSATION_VOICE(_EN/_ZH), AUDIO_CONVERSATION_SPEED, AUDIO_CONVERSATION_DIR, AUDIO_CONVERSATION_NO_PLAY, AUDIO_CONVERSATION_KEEP_FILES, TTS_GEN. Turn files keep the turn-NNN.mp3 naming convention.

Settings page

Settings → Guide Dog (id guide-dog):

• Auth — mmx auth status result with the key masked (sk-c…xxxx); never logged in full.
• Voice mode — global default on/off radios (per-session override lives on the small speaker button at the input's bottom-left).
• Voice input — STT engine select (whisper / sherpa / minimax), recognition language (auto/zh/en), input device select (defaults to the system default), and auto-send-after-recognition checkbox.
• STT — faster-whisper availability + version/python, and the whisper model select (base/small).
• Speak tester — text + voice selector (from guide-dog/voices), plays the mp3 in the browser.

Phase 1 — voice mode & voice input

Feature list

• Voice mode (host event-driven) — a host session/event listener watches assistant/message events, extracts the reply text (event.data.content blocks with type === 'text'), checks whether voice mode is effective for that session (session override else global default), and enqueues the TTS result ({url, key}) or error into a per-session voiceQueue. The client polls the queue every second and plays it with a module-level Audio object, or shows a bottom-right toast + beep for 6s.
• Voice cluster — conversation.input.left entry guide-dog-voice (order 30) at the input box's bottom-left, themed with DSH tokens (--dsw-alias-*), inheriting the app font:
  • small speaker icon — click toggles the per-session voice-mode override (guide-dog/set-config with voiceMode.sessions); hover tooltip shows "Voice mode: on/off · Global default: on/off".
  • language dropdown — recognition language detection (auto/zh/en).
  • mic icon — record → transcribe → insert (feather-style SVG; recording state pulses red with a second counter).
• Session-scoped playback — playback runs on a module-level Audio object, so switching sessions never replays or interrupts it: the current clip plays to the end unless a new playback task (a fresh queue entry from any session) overrides it.
• Mic voice input — the mic in the cluster: MediaRecorder with 1s timeslices, live second counter, maxSeconds auto-stop, language from the dropdown, and transcribe via guide-dog/transcribe. Recognized text is inserted into the input box with inputActions.setDraft(text) (auto-send via inputActions.submit() when configured). Error states: mic_denied, no_device, empty_speech, stt_failed, stt_timeout, engine_unavailable, insert_failed (never silent).
• Recorder page — sandboxed clients that cannot record in-page get a 🎙 Open recorder page link to the standalone /guide-dog/recorder page (GET serves a self-contained HTML recorder; POST /guide-dog/transcribe-upload accepts raw audio/webm, 20 MB cap, and runs the same transcribeImpl).
• Settings controls — the Phase 1 config blocks above, backed by guide-dog/get-config / guide-dog/set-config / guide-dog/status.

config.json schema

Lives at ~/.dsh/guide-dog/.guide-dog/config.json (auto-created from defaults; all keys optional, deep-merged over the defaults):

{
  "voiceMode": { "default": false, "sessions": { "<sessionId>": true } },
  "voiceInput": {
    "autoSend": false,
    "engine": "whisper",
    "language": "auto",
    "maxSeconds": 60,
    "whisper": { "python": "python3", "model": "small" }
  },
  "tts": {
    "voiceEn": "English_expressive_narrator",
    "voiceZh": "Chinese (Mandarin)_Gentle_Youth",
    "speed": 0.95,
    "format": "mp3"
  }
}

• voiceMode.sessions maps a session id to a boolean override; default is the fallback. The speaker button at the input's bottom-left toggles the current session's override.
• voiceInput.engine: whisper (only engine implemented; sherpa/minimax are reserved — selecting them returns engine_unavailable).
• voiceInput.maxSeconds forces the mic recording to stop.

STT engine (faster-whisper)

The whisper engine shells out to a bundled Python script (.guide-dog/scripts/whisper_transcribe.py) using faster-whisper:

pip install faster-whisper        # needs Python 3.8+; installs torch cpu wheels
python3 -c "import faster_whisper; print(faster_whisper.__version__)"

The host probes availability at startup and writes the result to .guide-dog/status.json (whisperAvailable, whisperVersion, whisperPython), shown in the Settings → STT row. Model choices: base (fast) / small (accurate); first run downloads the model weights.

Verification

node --check plugin-host.js && node --check plugin-client.js          # syntax
curl -s http://127.0.0.1:3080/guide-dog/recorder | head -5             # recorder page serves HTML
curl -s -X POST http://127.0.0.1:3080/guide-dog/api/guide-dog/status \
  -H 'content-type: application/json' -d '{}' | head -5               # status RPC (compat layer)
cat ~/.dsh/guide-dog/.guide-dog/status.json                            # whisper probe result

Manual checks (after deploy): click the speaker button (voice mode on, turns green) → send a message → the assistant reply is spoken automatically; switch sessions mid-playback → the clip continues to the end and is NOT replayed; use the mic button → recognized text appears in the input box; Settings → Guide Dog shows the Voice mode / Voice input / STT blocks.

Phase 2 — call mode

Feature list

• WebSocket-free dual channel — uplink is a whole-clip POST /guide-dog/call-transcribe (webm/opus, ≤20MB, reuses the Phase 1 transcribeImpl and the local whisper pipeline) → {ok, text, language, durationMs}; downlink is GET /guide-dog/tts-stream over a chunked HTTP stream (the host spawns mmx speech synthesize --stream --format pcm --sample-rate 24000 per sentence and pipes stdout incrementally into res.write; the client reads the stream with fetch().body.getReader() → PCM→WAV → seamless Web Audio scheduling). No new WebSocket protocol surface on the transport layer; browser and CLI reuse the same pipeline.
• Automatic VAD + push-to-talk (PTT) — default VAD (call.mode='vad'): MediaRecorder (audio/webm;codecs=opus, 250ms timeslice, continuous recording) + a parallel AnalyserNode energy detector (RMS ≥ vad.threshold; silence for vad.silenceMs ends an utterance; vad.minSpeechMs minimum speech; vad.maxSegmentSeconds per-segment cap) — speak-pause-speak automatically becomes two turns; the panel can switch to ptt push-to-talk (hold the mic to talk, release to send; VAD parameters do not participate in endpointing, only in interruption monitoring).
• Consensus-first (core interaction paradigm) — active only when call/a11y is on; typing mode keeps the Phase 1 behavior: a prompt soft constraint (guide_dog_call_consensus systemPrompt variable, conversational wording: understand intent first, ask when unclear, explain before writing/modifying and wait for the user's go-ahead) plus a mechanical hard guarantee (tools/pre-execute waterfall interception: write/edit and destructive-bash heuristics rm/mv/cp/truncate/dd/overwriting-redirect/git push etc. → unconfirmed returns {kind:'deny', reason: 'needs_voice_confirmation'} and the model asks by voice; user confirmation keywords hit → released for this turn; before every execution the host TTS-broadcasts a one-sentence summary built from the tool args (not through the model), then opens a consensus.summaryWindowMs interruption window; speech inside the window aborts the execution — the tool has physically not started). Interceptor failure → deny and announce "consensus check failed" (better to block wrongly than to allow wrongly, spec §6.8).
• Progress announcements (minimalist principle, RC10) — only useful information is announced: agent/status (running → "processing"), tools/result (tool name → phrase: write/edit → "modifying files", web_search → "searching the web", guide_dog_image/video/music/speak → "generating media", bash only for destructive commands (the same DESTRUCTIVE_BASH_RE as consensus) → "running a command"; read/grep/glob/ skill/non-destructive bash/unknown tools stay silent), agent/error ("processing failed: "); same-phrase 4s cooldown dedupe (multi-step same-kind operations announced once); >120s without any event during a call → heartbeat "still processing, please wait". Announcements and reply playback share one queue: announcements first (queue head), replies yield; announcements go through the streaming channel (the same WebAudio PCM chain as replies) → constructive serialization on a single player, one after another — overlap is impossible.
• Streaming TTS — reply text is split per sentence (stream.sentenceSplit charset 。！？.!?\n; stream.maxSentenceChars force-truncates over-long sentences) and synthesized sentence by sentence; each sentence gets a fresh one-time token via guide-dog/tts-token (single-use, 5-minute validity, bound to sessionId); pre-synthesis between sentences (while the current sentence plays, the client requests the next sentence's stream ahead of time and appends decoded frames seamlessly on the playback-time chain — the next sentence continues before the previous one finishes; long replies are read in full with ≤400ms gaps). Measured ~600ms to first byte for short Chinese sentences, satisfying the "first audio <1.5s" criterion. Only the turn's final message is played (RC13): intermediate assistant messages (with tool-call blocks) are not enqueued — playing near-identical text per step was the root cause of "the same content repeated"; intermediate steps are covered by progress announcements. A terminal tool turn (the last message still has tool calls) is covered by the turn/end fallback that plays the buffered text — never silent.
• Barge-in — VAD detects user speech during playback (≥ vad.interruptMinMs 300ms to avoid false triggers) → the browser immediately stops playback and clears the unplayed buffer → the stop is a 10ms fade-out (RC13) — src.stop() hard cuts click at sentence boundaries. The first transcript segment after the interruption goes straight to the current turn via the interrupt RPC (agent.steer, RC11) instead of queueing as a new turn → abort the current tts-stream fetch → the new speech naturally becomes the next turn (Pipecat InterruptionFrame semantics).
• Voice commands — call transcriptions that hit the command table (pause, resume, repeat/say-again, slower/faster, look-at-screen [Phase 3 stub]) execute locally and are not submitted to the conversation (stop/continue are consensus confirmation words, not in the command table — they pass through to the agent unchanged); guide-dog/call-command RPC provides host-side commands such as clear-queue.
• Dual-channel mutual exclusion (RC13) — text already spoken on the host speakers via guide_dog_speak(playOnHost=true) is not replayed through the voice-mode/call queue channels (consumed on use, the same text blocked once) — eliminates the "host + browser double sound".
• Fault tolerance — a stream interruption auto-reconnects once (at most one retry per (sid,text) within 5s, no retry on 429; a fresh token per sentence; failure toast "playback interrupted"); STT failure does not submit + beep + toast; TTS failure still lands the text + failure beep + panel error state (never silent); consensus-interceptor failure denies conservatively and announces the reason. Session ownership for call transcription/interruption/polling is captured once when the call starts (RC13) — multi-session switching no longer cross-talks.

RC14 fixes (2026-08-17): announcement content selection + queue tail-truncation + progress dedupe + double-play pinpointing

• Announcement sanitization (sanitizeSpeechText, F1) — before enqueueing, reply text goes through a markdown/URL/emoji strip: [title](url) keeps the title, drops the URL; bare URLs (https?://, www.) are removed entirely; leading list/quote markers (-/+/*/>) and leading ordered-list markers (1. 1、 1)) are stripped; markdown markers such as **bold** and backticks are stripped; emoji ranges (U+1F000-U+1FAFF etc.) are stripped. Call mode reads only human language — no URLs/**/-/📢 metacharacters, so URL fragments like "thepaper/newsD/weather.com" are no longer read out in pieces.
• Smart sentence splitting (splitSentences, F2) — in the default separators '。！？.!?\n', . follows a smart rule: split only when . is followed by whitespace + an uppercase letter/digit/CJK ('Hello. Next' → 2 sentences; '8.17 的上海' stays 1; dots inside URLs are never split). The Chinese separator set gains ；; so …； no longer cuts one Chinese reply into two halves.
• Queue cap 40 with tail truncation (VOICE_QUEUE_MAX, F3) — raised 10→40, drop from the tail to keep content: on overflow while (q.length > VOICE_QUEUE_MAX) q.pop() (first-in content wins; the old splice(0, …) head-removal strategy cut the main content first while keeping URL fragments). announce/hb progress still use pop() (unshifted to the queue head, progress first).
• 30s progress-phrase dedupe window (F4) — announce's progressDedupe cooldown extended 4s→30s: web_search results ~4.3s apart no longer announce "searching the web" three times. The progressDedupe function body is untouched; repro-progress.js semantics preserved.
• Dual-channel mutual exclusion by sanitized text (F5) — wasHostSpoken / markHostSpoken uniformly use sanitizeSpeechText-processed text as the key: all three wasHostSpoken call sites (downlink, turn-end flush, voice-mode) match on the sanitized key; speakImpl registers both the raw and the sanitized key after a successful playOnHost (double key) so any downstream channel dedupes correctly. Known edge: when transform.py rewrites the text the two keys may differ slightly (acceptable edge).
• Diagnostic instrumentation (F6, one-shot retest to pinpoint "reading twice") — zero behavior change, logs only:
  • host ([gd-host], visible in the DSH terminal): enqueue from=downlink|turnend|voice-mode|consensus|announce|heartbeat n=... qlen=..., shift key=... remain=..., skip host-spoken sid=... text=..., QUEUE-DUP text=....
  • client ([gd], browser DevTools): playStreamEntry ... times=... (accumulates per entry.key || entry.text on each play), PLAY-SUMMARY key=N | ... (when the queue empties, summarizes all current counts then clears).
  • Retest calibration (basis for the RC15 direction): QUEUE-DUP → host double-enqueue; PLAY-SUMMARY key=2 → client double-play; neither but still twice → tts-stream double audio write; enqueue from= same source twice with the same text → event replay.

RC15 fixes (2026-08-17): persistent player + gesture unlock + failed-entry requeue + event-replay dedupe