dsh-sensevoice-npu

dsh-sensevoice-npu

Local speech-to-text for DSH that runs SenseVoiceSmall on the Intel NPU through OpenVINO, plugged into the official voice-input stack.

It registers one more provider into the stock @deepseek-ai/dsh-experimental-speech-to-text service, so the voice-input button, the settings card, the preparation UI and the speech API keep working unchanged — only the recognizer behind them changes.

Why a separate plugin

The bundled provider (dsh-experimental-speech-to-text-sensevoice) is CPU-only by construction:

• lib/worker.js hard-codes provider: "cpu" for the recognizer and the VAD;
• the shipped sherpa-onnx-win-x64 package is a CPU-only ONNX Runtime build, so sherpa-onnx's openvino provider logs "OpenVINOExecutionProvider is not available … Fallback to cpu";
• ONNX Runtime publishes no OpenVINO build for Windows;
• and OpenVINO crashes the NPU compiler on this model's default graph (unbounded dynamic dimensions from the x_length attention mask).

This plugin sidesteps all four: it bakes x_length into a constant for one frame bucket (which the NPU compiler accepts), compiles that graph for the NPU, and serves it through a small Python worker.

Layout

The frontend mirrors sherpa-onnx exactly: kaldi_native_fbank with the same options, ApplyLfr (7/6), and (x + neg_mean) * inv_stddev with the CMVN vectors read from the ONNX metadata — so transcripts match the CPU provider.

Install

Prerequisite: model files

The worker reads three files (this plugin does not download models — it registers with an empty downloadSources):

The official dsh-experimental-speech-to-text-sensevoice provider downloads them into the same dataRoot (<DSH home>/speech-to-text/sensevoice), so let that provider prepare once, or place the files at those paths yourself.

1. Build the Python environment (needs a 64-bit Python 3.11–3.13 and the Intel NPU driver):

   pwsh -File runtime/bootstrap.ps1
   

2. Install the package into the profile that owns the voice-input bundle (the DSH desktop app uses the desktop profile):

   dsh plugin --profile desktop add "file:C:/path/to/dsh-sensevoice-npu"
   

   pnpm copies the package, so while iterating on it replace the copy with a junction to your working tree:

   $d = "$env:USERPROFILE\.dsh\profiles\desktop\node_modules\dsh-sensevoice-npu"
   Remove-Item -Recurse -Force $d
   New-Item -ItemType Junction -Path $d -Target C:\path\to\dsh-sensevoice-npu
   

3. The bundle's own cordis.patch.yml already selects the provider and registers the plugin, and it contains no machine-specific path: pythonPath and dataRoot default to the bootstrap.ps1 venv and DSH's speech root. Append an override to ~/.dsh/profiles/desktop/cordis.patch.yml (back it up first) only when you moved either one:

   - id: speech-to-text-npu
     name: 'dsh-sensevoice-npu'
     config:
       pythonPath: <custom venv>\Scripts\python.exe   # only with -VenvDir
       dataRoot: <custom speech root>                 # only with a non-default DSH_HOME
   

4. Restart DSH. The voice-input card then lists SenseVoiceSmall · Intel NPU (NPU) and uses it by default; the first preparation compiles the NPU graph once (~2.5 min), after which starts take ~0.2 s. If the card still shows the CPU provider, pick the NPU one in the voice-input settings (an explicit choice persists over the patch default).

Configuration

Each bucket costs roughly 800 MB on disk (baked IR + OpenVINO NPU cache) and one ~2.5 minute compile the first time. One bucket is enough for dictation: 256 frames ≈ 15 s of audio, and the NPU handles the padding cheaply.

Measured behaviour (Core Ultra X7 358H, Arc B390, NPU driver 32.0.100.5540)

CPU is used only for the frontend (fbank + VAD, a few ms) and JSON plumbing; the encoder/decoder graph runs on the NPU. Note that the NPU is not faster than sherpa-onnx on CPU for clips this short (~0.5 s vs ~0.2 s) — the win is moving the heavy work off the CPU/GPU entirely.

Development

node --test        # 6 tests, no dependencies

They cover the registration contract (provider id, device label, diagnostic file), the portable pythonPath/dataRoot defaults, and the two config guards that would otherwise fail much later: an empty bucket list and a relative path.

Troubleshooting

• Confirm the profile really loaded the plugin and see the last phase/device:

  Get-Content "$env:USERPROFILE\.dsh\speech-to-text\sensevoice\npu\provider-status.json"
  

• NPU device not visible → update the Intel NPU driver (32.0.100.5540 tested).

• NPU compile failed → delete <dataRoot>/npu and retry; check free disk (~1 GB per bucket).

• Transcription slow on first use of a new bucket → it is compiling; later runs reuse the cache.