dsh-video-storyboard

dsh-video-storyboard

A scene-aware storyboard for text-only agents. Feed a video in, get one addressable grid out: numbered cells, a timestamp per cell, and the transcript line that falls inside it — so the model can *look*, *cite a cell*, and *ask for a close-up* instead of being told a summary.

The effect, in numbers. A 5-minute clip becomes one image — 9–16 frames for 384 tokens, where a frame-per-image tool spends 3,072–6,144 for the same coverage. A 90-second PV scans in 9 s and returns 12 cells whose shot boundaries match 41 detected cuts. Second call on the same file: 3.6 s (cached). No model weights, no image library, no font config.

Community project. Not affiliated with DeepSeek.

中文说明：README.zh.md — 创新点、实现原理、实测基线、路线图。

Requires: Node ≥ 20, ffmpeg/ffprobe on PATH. No other runtime dependencies.

*A 27-second synthetic clip (9 shots, loud/quiet/loud audio) turned into one image: 9 numbered cells, one timestamp range each, and a footer strip showing loudness, silence and cut points on the same time axis. The clip is generated by ffmpeg — no third-party footage is used anywhere in this repo.*

*The same tool on the author's own 19-minute video: cells 17–32 of a 64-cell run (4 images). The yellow ticks on the footer are the 408 cut points found in the film. This image is the author's footage, used as documentation only — see NOTICE.*

Where the design came from

Three instincts, each of which became a mechanism (and two of which needed the wording tightened once the code existed):

intuition · what it became "read the transcript first, then decide how to slice" · audio and transcript are computed before any cell decision; chars/sec is a third density signal. Precisely: the transcript decides *how long each cell is*, not *where the cuts are* — cuts stay a visual signal "pick the sharpest frame, do not get fooled by a blur transition" · 3 candidates per segment, highest Laplacian variance. ffmpeg thumbnail picks by histogram, and blur does not change a histogram "give the fast part more cells, but do not let a half-second shot eat a whole cell" · dense stretches get short cells; minGap is not a fixed 1.2 s — it is capped by the material's own granularity (≤ 0.9 × median shot), otherwise six 1-second shots merge into three and the grid cannot be filled

What the algorithm actually does

Eight decisions, each made because the obvious alternative failed on real footage:

# · Decision · Why not the obvious thing · Code 1 · Scene-change detection on HSV at 64×36, three-channel mean of \ · ΔH\ · +\ · ΔS\ · +\ · ΔV\ · · Luma-only difference misses cuts that swap hue at equal brightness (2 of 9 cuts in our fixture; ffmpeg scdet finds 4/9, we find 9/9) · src/scan.js 2 · Threshold chosen by rate, not by a score multiple — target ≈1 cut per 1.5 s, floor at score 12 · 3×p95 scored 38 on B-roll (fine) but 130 on fast-cut anime → zero cuts, on exactly the content that needs the most cells · src/scan.js 3 · Cell placement = equal increments of the local cut-rate integral, snapped to real cuts (≤30 % of an ideal cell) · Uniform splitting gives a 6 s burst of 20 shots the same space as a 20 s static shot; snapping without a limit silently distorts the weights · src/density.js 4 · Grid size from motion p90, not from duration · 16 cells over a static single take = 16 identical frames; 4 cells over a PV = nothing · src/plan.js 5 · Minimum gap capped by the material's own granularity (min(minGap, medianShot×0.9)) · A fixed 1.2 s merged six 1 s shots into three, and then could not reach the requested cell count at all · src/segments.js 6 · Sharpest frame wins — 3 candidates per segment, highest Laplacian variance · thumbnail=n= picks by histogram; blur does not change a histogram, so it happily returns a Gaussian-smeared credit card · src/sheet.js 7 · Audio envelope on the same axis, inside the same image · A separate audio image costs another 384 tokens and loses the alignment, which is the entire point · src/audio.js, src/sheet.js 8 · ASR chunked to ≤45 s at silence midpoints, non-speech segments filtered and reported · One 299 s request shifted every timestamp 13 s early; a music-only PV produced a fake 29 s line that landed in 5 cells · src/asr.js, src/pipeline.js

Reading order is a choice, and one of them spoils the film

An overview image of a 2 h film puts the last scene on the same screen as the first. That is fine for "what is this", and fatal for "what happens next".

· what it does · when STRUCTURE — cells: 16, sheets: 4 over the whole file · 4 images, whole arc at once. ~310 s CPU for 2.5 h · analysing, indexing, "is this worth watching" SEQUENCE — from/to windows of 2–5 min, one at a time · never shows a later scene than the one you asked for · following a plot, suspense, shot-by-shot study

video_probe returns both, labelled, and refuses to pick for you. Nothing in the tool forces a whole-file pass: from/to is the primary interface, and each window's scan is cached so re-reading a stretch after the first pass costs nothing.

The two-stage read, and what a long film actually costs

A 2 h 35 m film is not 24 images up front (9,216 tokens). The cheap path is:

stage · cost · what you learn video_probe · 0 · length, audio/subtitle tracks, how to read it one overview (16 cells over the whole file) · 384 · structure — which 7.5-minute stretch is worth opening open the 2–3 suspicious stretches · 3 × 384 · 16 cells of detail each total · 1,536 · same price as four images, but the first 384 already tells you whether to continue

The saving is not in shrinking any single image — it is in letting the model stop early. That is why video_probe returns a plan rather than a picture, and why textOnly paging exists.

Prior art, accuracy ceilings we do not try to beat, and the things we deliberately left out: docs/REFERENCES.md.

Long films: the text index has a budget

One image is 384 tokens, but the *text* index is unbounded — a 2 h 35 min film produced 64 cells and a 75 KB index, which exceeded the host's tool-result limit. The excess was folded into a file on disk and never read, so 44 minutes of dialogue were silently missing. Three things prevent that now:

• Per-cell cap, three levels: detail: low 60 / med 120 (default) / high 400 characters, and the

cut lands on a sentence boundary — a budget of 120 stops at the last 。 inside the window rather than mid-word. Measured on 9 cells: 1,521 / 1,915 / 3,031 characters.

• Cell-aligned truncation with an explicit continuation hint:

[text stopped at cell 13 of 64 — call again with cellRange: [14, 64] and textOnly: true to read the rest]

• `cellRange` + `textOnly` — page through the index without re-rendering images or re-decoding frames.

video_storyboard({ input, cells: 16, sheets: 4 })                     // 4 images + first page of text
video_storyboard({ input, cells: 16, sheets: 4, cellRange: [14, 42], textOnly: true })
video_storyboard({ input, cells: 16, sheets: 4, cellRange: [43, 64], textOnly: true })

textOnly returns no image, so paging costs nothing in tokens; with the scan and ASR results cached it takes ~0.8 s per page after the first call.

CLI batch mode for a whole film, where 24 separate calls is not a workflow:

node bin/storyboard.mjs film.mp4 --batch 300 --out film.png
# → film-w01.png … film-w24.png  +  film-timeline.md (window | cells | cuts | chars | image)

A tail window under 30 s is merged into the previous one instead of spending 384 tokens on a 2-second window that holds a single cell. Window boundaries are snapped to real cuts (one full scan up front, --no-snap to disable), so a window never ends in the middle of a sentence.

Full install & usage guide: docs/USAGE.md · 中文：docs/USAGE.zh.md — every parameter, six workflows, troubleshooting, and what each one costs.

Quick start

git clone https://github.com/SAOS-3624/dsh-video-storyboard.git
cd dsh-video-storyboard

node --test                                          # 154 tests (needs ffmpeg)
node bin/storyboard.mjs your-video.mp4 --cells 9 --text   # see exactly what the model would get
node bin/storyboard.mjs film.mp4 --batch 300              # long film: one png per 5-min window + a timeline.md
node scripts/install-dev.mjs                         # mount into DSH, then restart DSH

Installing into DSH. For a normal profile:

dsh plugin --profile web add dsh-video-storyboard

The desktop profile is managed exclusively by the Electron app, so dsh plugin refuses it — use node scripts/install-dev.mjs instead (it copies the package into the profile and registers the bundle). Either way, restart DSH afterwards: the plugin tree is composed at process start.

How sampling actually works (and what it is not)

Worth stating plainly, because it is easy to misread:

• Tokens are billed per image, not per frame. Everything here exists to put *many* frames into

*one* image. 9–16 frames = 384 tokens.

• All cells are the same size. This tool does not shrink resolution in fast-cut sections.

What it varies is how many seconds each cell covers — dense regions get short cells, calm regions get long ones (density(t) = clamp(localCutRate / meanCutRate, 1, 2.5)). Why not variable resolution: at 350×194 a halved cell is mush, and the token cost is flat anyway.

• The timeline is fully covered; individual shots are not guaranteed. 300 shots in 5 minutes across

16 cells cannot all be seen. That is what video_zoom is for.

• Continuous motion is not a cut. Any frame-difference scorer (this one, scdet, TransNetV2)

gives high scores to a fast pan or a zoom. That is why peaks are suppressed within 0.35 s and why short segments get merged instead of each taking a cell.

Status (v0.1)

Piece · State src/plan.js — layout / coverage / grouping / guardrails (pure functions) · ✅ implemented src/scan.js — HSV-weighted shot boundary detection · ✅ implemented — 9/9 cuts, 0 false positives on the fixture, 47× realtime src/segments.js — cuts → segments → fit to N cells · ✅ implemented src/probe.js — ffprobe + rotation / end-edge handling · ✅ implemented src/sheet.js — extract + cover-crop + burn index + tile, zero image library · ✅ implemented adaptive threshold + min-gap + density-driven grid sizing · ✅ validated on 6 real clips src/pipeline.js — orchestration shared by CLI and tool · ✅ implemented src/index.js — registers video_storyboard + video_zoom · ✅ implemented, host contract verified src/audio.js — loudness (EBU R128) + silence, no model · ✅ implemented src/asr.js — subtitle track + OpenAI-compatible ASR + cell alignment · ✅ implemented (client verified against a local fake server) src/cache.js — scan-result cache keyed by file+window+threshold · ✅ implemented (2nd call on a 5-min clip: 13.4 s → 3.6 s)

npm test                                                  # 154 passing (unit + ffmpeg e2e + host contract)
node bin/scan-video.mjs video.mp4 9                       # 调阈值：切点 / 分数 / 段长 / top peaks
node bin/storyboard.mjs video.mp4 --cells 9               # 端到端出一张图 + manifest
node bin/storyboard.mjs film.mp4 --from 600 --to 780      # 只看某一段（不扫全片）
node bin/storyboard.mjs film.mp4 --plan 300               # 先出分段计划（不渲染）

Against ffmpeg -vf scdet on the same clip: scdet's default threshold finds 4 of 9 hard cuts; this scanner finds 9 of 9 with a 2× score margin to the strongest non-cut frame. Reason: scdet scores mean absolute *frame* difference (luma-ish), so a cut that swaps hue at equal brightness is invisible to it. Measured on the two such cuts in the fixture: brightness delta 0.6–0.8 %, hue delta 24/255 and 151/255.

Real-world check (6 clips of 1080p footage: B-roll, anime, PVs, a single take)

clip · len · p90 · auto th · cuts · gap · grid B-roll (50 mm lens) · 299 s · 7.9 · 16 · 141 · 2.1 s · 16 ×1 anime feature · 1090 s · 15.0 · 43 · 408 · 2.7 s · 16 ×2 special ED · 80 s · 11.6 · 21 · 42 · 1.9 s · 4 ×1 PV (official) · 92 s · 17.0 · 54 · 44 · 2.1 s · 9 ×1 PV (concept) · 90 s · 26.1 · 58 · 41 · 2.2 s · 9 ×1 party singing (single take) · 251 s · 2.2 · 12 · 1 · — · 4 ×1

The threshold is chosen by rate, not by a score multiple. My first attempt was 3 × p95, which looked fine on the B-roll clip (38 → 88 cuts) and then died on fast-cut anime: the concept PV's p95 is 43, so ×3 = 130 → zero cuts, on the exact content that needs the most cells. Targeting ≈1 cut per 1.5 s (auto-tightened to 0.5 s on short clips, floored at score 12 so static footage stays a single shot) works across all six.

Grid size follows content density (p90), not duration alone: ≥15 → 12 s/cell, 5–15 → 20 s/cell, <5 → 40 s/cell. A fast-cut PV gets 9 cells over 90 s; a static single take gets 4 cells and a warning.

Minimum gap between sampled frames is enforced by merging segments shorter than minGap (default 1.2 s) before fitting to the cell count — a 0.3 s shot in a burst should not eat a whole cell.

Pick the sharpest frame, not the blurriest

--pick sharp (default) samples 3 candidates inside each segment and keeps the one with the highest Laplacian variance. On the credit card of a PV, --pick thumb returned a Gaussian-blurred frame (text unreadable) while --pick sharp returned the crisp one — same segment, same second.

Cost on a 90 s clip / 9 cells: mid 8.0 s · `sharp` 9.2 s · thumb 11.3 s. Sharp is *cheaper* than thumbnail=n= because three seeks decode three frames, while the thumbnail filter decodes the whole segment.

Read a slice, not the whole file

--from 600 --to 780 passes -ss/-t to ffmpeg, so scanning and sampling only touch that window: an 18-minute film sliced into 3 minutes takes 15.8 s end to end (vs 22.7 s just to scan the whole thing). This is how the model is meant to read a feature film — five minutes at a time, on demand.

Audio: a strip inside the same image

The sheet has a footer strip on the same time axis as the cells: teal bars are momentary loudness (EBU R128, measured locally with ffmpeg), grey blocks are silence ( 0.6 s, drawn as a backdrop *behind* the bars), yellow ticks are cut points. Note the bars are normalised against p10–p97, so a short bar means "quiet for this film", not "silent" — only the grey backdrop means silence.

The strip is a *rough* locator — at 1000 px for 300 s one pixel is 0.3 s, and a vision encoder may downscale it further, so read a tick to ±2–4 s. Exact times live in the text: each cell's range is given to 0.01 s, and a window with ≤ 24 cuts lists every cut point individually. So the model can say "cell 5 sits in the quiet stretch before the swell" — an alignment neither a music-analysis plugin nor a transcript alone can give it.

It costs no extra tokens (same image), only ~9 % of the pixel budget: cells are laid out at budget × 0.9 and the strip takes the rest. audioStrip: false turns it off.

What it does not do: say *why* the music sounds like that. Loudness is measurement, not understanding — see the design doc for why (a mastered track sits inside a 3 dB band, so the envelope alone can't find the chorus).

Caching

Scan results are cached by (path, size, mtime, window, threshold) — video_storyboard followed by video_zoom on the same clip scans once. Loudness is not cached (2 h at 10 Hz is ~70 k floats, and audio decode is cheap compared to video). Cap 50 MB with LRU eviction; DSH_VIDEO_CACHE_DIR points it at another drive; cache: false disables it.

Note on long videos: decoding dominates (~25 s for a 1090 s 1080p film), and decimating the scan does not help — measured 24.9 s at full rate vs 22.5 s at 2 fps, because the cost is H.264 decode, not the HSV scoring. -skip_frame nokey is 5× faster but produces frames that differ from a normal decode, so it is unusable. For long films, use --from/--to or --plan.

Transcript: subtitles first, then ASR, never fatal

1. Embedded subtitle track → extracted with ffmpeg (free, exact).
2. No subtitle + DSH_ASR_KEY configured → OpenAI-compatible /audio/transcriptions

(default FunAudioLLM/SenseVoiceSmall), response_format=verbose_json for timestamps.

1. Neither → the sheet still renders; the text block says transcript="no-audio" / asr-failed.

Each cell then carries the lines that overlap it:

cell 5 (r2c2, image 1) = 00:01:12.40-00:01:22.80  frame@00:01:17.20
    "我觉得这个好像和我想要的东西有点接近"

Music is not speech, and ASR will invent words for it. On a music-driven PV (90 s, one real line) one request produced a fake 29 s segment (45.35–74.33, 9 characters). Because cells are filled by time overlap, that single fake line landed in 5 cells — the same garbage quoted five times, which is worse than silence. filterSuspectSegments drops segments that are either punctuation-only or longer than 8 s with under 0.8 chars/s (real speech is 2–6), and the text block reports how many were ignored so the filter never hides data.