dsh-plugin-voz-copilot

dsh-plugin-voz-copilot

Voice copilot for DeepSeek Harness. It gives the push-to-talk session a name and a place in the sidebar, adds dictation microphones to the composer, and can run a real-time spoken conversation: the microphone stays open, a Silero VAD decides when you speak, transcription starts while you are still silent, and the answer is spoken back sentence by sentence while the model is still writing.

The plugin is the glue between the harness and a speech stack running on localhost. It does not ship a speech-to-text engine or a text-to-speech voice; those are external services (see Requirements).

Install

npm install -g pnpm
npx @deepseek-ai/dsh plugin --profile web add github:songjongdevsecops/dsh-plugin-voz-copilot --allow-build=onnxruntime-node

--allow-build=onnxruntime-node é obrigatório no pnpm 10+: sem ele o pnpm bloqueia o download do binário do ONNX Runtime e a instalação falha. O pacote em si não tem passo de build.

restart the harness (stop dsh web and start it again)

npx @deepseek-ai/dsh web

The package declares `dsh.bundle.patch`, so `dsh plugin add` installs it **and**
adds its line to the profile bundle list. The bundle layer is read at boot, so a
restart is what activates it.

If `dsh` is installed globally, use `dsh` instead of `npx @deepseek-ai/dsh`.

### Remove

```sh
npx @deepseek-ai/dsh plugin --profile web remove dsh-plugin-voz-copilot
# restart the harness

Removing the package removes its bundle line. Conversations already named keep their title in the session log; nothing is deleted.

Requirements

• Node.js ^22.19.0 || >=24.0.0 (the same range DeepSeek Harness supports).
• pnpm on PATH (the harness manages profile plugins with it).
• DeepSeek Harness with the web profile for the interface. The title mode is used by the acp profile.
• Speech-to-text: an HTTP endpoint compatible with whisper.cpp's whisper-server, at whisperUrl (default http://127.0.0.1:8178): POST /inference?response_format=json with the audio in the multipart field file, answering {"text": "..."}. The server must decode WebM/Opus (an ffmpeg-enabled build), WAV and MP3. This plugin does not install it.
• Optional fast engine: whisperFastUrl (for example http://127.0.0.1:8179) for short conversation segments. When it is not set, the main endpoint is used.
• Optional text-to-speech: the edge-tts CLI, or a warm HTTP server. See Configuration. Without tts, the real-time conversation button stays disabled with the reason in its tooltip; dictation still works.
• Optional speaker verification: Python 3 with sherpa-onnx and numpy (requirements.txt). The ERes2Net model (38 MB, Apache-2.0) and the Silero VAD model (2.3 MB, MIT) are bundled. The speech detector uses onnxruntime-node, installed by npm.

Use

The plugin is additive: it does not replace any harness control.

• Push-to-talk session. With a companion push-to-talk bridge that binds the Copilot key (F23 in the reference setup) and opens the session with [conversa_copilot] as the first line of the turn, this plugin pins the session title to conversa_copilot and attaches the session to the configured workspace, so it appears in the sidebar instead of staying invisible. The bridge and the key binding are not part of this package.

• Header microphone. Binds the conversation that is open in the GUI as the target of the voice. Lit means this is the voice conversation; a second click unbinds it. Binding writes the session id to statePath (the same file the bridge reads) and attaches the session to the workspace.

• Sidebar footer. Shows which conversation is receiving the voice without opening it.

• Composer microphones. Two buttons next to Send:

  • the plain microphone records and leaves the transcription in the draft, so you can edit before sending;
  • the microphone with an arrow records and sends the turn directly.

  Click the same button again to stop. While recording, the button becomes a red stop square; Esc or the cancel control in the floating pill aborts. The transcription is appended after whatever was already in the draft, never replacing it. An empty transcription is an error, never an empty turn.

• Real-time conversation. The third button opens the spoken loop on the open conversation: speak normally, the answer is spoken while it is written, and speaking over it interrupts (barge-in). The panel has pause, stop-response and close controls; Esc closes it and always releases the microphone.

• Only your voice. After enrolling your voice in the conversation panel, speaker embeddings decide whether a segment belongs to you. Audio from other voices is ignored before transcription. The check is fail-open: no enrollment, no local server, or too-short audio lets the speech through.

How it works

Two halves, one package:

The browser never talks to the speech servers directly: the host process owns the loopback route and enforces a local-request boundary (loopback Host, same-origin Origin, POST for state changes, 256 KB for JSON and 16 MB for audio). Audio and transcripts stay on localhost unless you point a configured service at a remote address.

Configuration

The defaults come from the package bundle patch and boot on any machine. To use your own workspace and voice, edit the profile patch (~/.dsh/profiles/web/cordis.patch.yml). A patch under the same id replaces the whole config of the line, so restate every key you want to keep.

- id: voz-copilot
  config:
    mode: 'attach'
    workspace: '/path/to/your/workspace'
    title: 'conversa_copilot'
    statePath: '~/.dsh/voz-copilot.json'
    pollMs: 3000
    whisperUrl: 'http://127.0.0.1:8178'
    whisperFastUrl: 'http://127.0.0.1:8179'
    voz:
      persona: 'Answer in short, plain sentences that sound natural when spoken aloud.'
      maxChars: 220
    tts:
      bin: 'edge-tts'
      voice: 'en-US-AriaNeural'
      rate: '+0%'
      volume: '+0%'
      lexicon: '/path/to/tts_lexicon.py'
      python: 'python3'
      servidorQuente: 'http://127.0.0.1:8180'
      servidorQuenteScript: '/path/to/tts-servidor.py'
      servidorQuentePython: 'python3'
    orador:
      limiar: 0.5
      perfilPath: '~/.dsh/voz-perfil.json'

~ is expanded in workspace, statePath and the other paths below.

Text-to-speech

tts.bin is spawned once per phrase with --voice, --text, and the optional --rate=, --volume=, --pitch=, and must write MP3 to stdout. With tts.servidorQuente set, the plugin first tries GET /fala?texto=...&voz=... on that server and starts tts.servidorQuenteScript when it is down, falling back to the CLI. The warm server is what makes the first byte fast; the CLI works without it.

Speaker verification

The default paths assume a checkout of this repository with a venv inside it:

git clone https://github.com/songjongdevsecops/dsh-plugin-voz-copilot.git
cd dsh-plugin-voz-copilot
python3 -m venv venv
venv/bin/pip install -r requirements.txt

Then install the plugin from the checkout (dsh plugin --profile web add /path/to/dsh-plugin-voz-copilot) so the bundled paths resolve, or point orador.python, orador.script and orador.modelo at that checkout. The Python server is optional: without it the conversation mode still works, just without speaker filtering.

Verify

# the host half is mounted and answering
curl -s http://127.0.0.1:3080/voz-copilot/api/estado
# expected: {"ok":true,"sessionId":null,"conversa":false,...}
#           "conversa" is true only when tts is configured

# the browser half is being served (revision of client.js)
curl -sN http://127.0.0.1:3080/plugins/events | grep voz-copilot

In the GUI: open a conversation, click the plain microphone, speak, and click it again. The text should appear in the composer. If the speech server is not reachable, the pill shows the error instead of silently doing nothing.

Limitations

• No speech engine included. Without a whisper-server at whisperUrl, dictation and conversation transcription fail with a clear error; the session naming/attach side still works if a bridge writes the marker and the state file.
• No voice included. TTS needs edge-tts (which sends the text to Microsoft's service) or your own warm server. Without tts, spoken replies and the conversation button are unavailable.
• Tested on Linux. Windows and macOS are untested. onnxruntime-node ships a native binary per platform; the Python venv path is platform-aware, but the external services are assumed to be POSIX-friendly 127.0.0.1 processes. The browser half is tested with Chromium (MediaRecorder, audio/webm;codecs=opus, AudioContext).
• Speaker verification is best effort. The threshold is a starting point; calibrate orador.limiar for your microphone and room. It is fail-open by design: a failure never drops your speech, so a missing server means every voice in the room can become a turn.
• Barge-in is a heuristic. A loud cough can cancel a response, and a cancelled response is not recovered.
• Caps are fixed. 30 minutes per recording, 16 MB per audio request. The conversation segment is cut at 10 seconds to fit a 12-second audio context; the text is joined instead of sending a turn mid-sentence.
• Latency depends on your machine. The transcription and TTS path is local HTTP plus a warm process; numbers from the reference setup are not a promise.
• One writer for the state file. statePath is shared with the bridge. If you run both, let the bridge own the voice link; binding from the GUI is for setups without the bridge.
• The title mode needs the acp profile. The web profile does not have the session title service used by that mode.
• The interface text is in Brazilian Portuguese. Button tooltips, the recording pill and the conversation panel are localized in Portuguese; the package documentation is in English. The behavior is the same either way.

License

MIT