dsh-session-vacuum

dsh-session-vacuum

Topics: dsh · dsh-plugin · deepseek-harness · session-cleanup · session-log · streaming-fragments · vacuum · prune · oom · long-session · memory-optimization

Is your DSH session getting slower every turn, slower to load, and eventually unopenable — or does it just OOM?

That is not a feeling. Every token is written twice: the fragments (assistant/chunk and the packed text-chunks / reasoning-chunks / tool-call-chunks rows) are read by nothing, and they pile up. A 1,813-turn conversation reached 272 MB on disk, expanded into 8.43 million objects when loaded, took 1.29 GB resident, and died.

This plugin deletes them on disk — not hidden, actually deleted — so the next load is smaller before anything else happens. Same session after one pass: 86.67 MB → 16.99 MB, 169,795 fragments → 0, and the non-fragment content: nothing lost (dropped=0).

You are accumulating this from turn one. Measured on real sessions, the number of prunable fragments tracks output tokens at roughly 0.95–0.99 : 1 — eight sessions, from 88 fragments for a 91-token one-turn chat up to 1,498 fragments for a 1,516-token six-turn one. Nothing reads them; they are write-only.

So you do not have to be in trouble yet. This is a preventive plugin as much as a rescue one: install it while the log is still small and the threshold is never crossed. It is not only about avoiding the crash, either — measured across 60 sessions, the median log-read speedup after a pass is ×7.77 (best ×97.5, scope spelled out below), which is exactly the cost a page refresh sits on. Below the thresholds it touches nothing, and a session you are using is never touched at all. Every DSH session accumulates this; none of them has to be broken first.

It runs unless you stop it, but you can look before it touches anything: dryRun: true records what it would do and writes nothing. A pass that cannot verify its own output writes nothing at all. The swap is atomic, and .orig is never deleted for you.

• Live by default. A fresh install prunes idle logs with no configuration at all. Set enabled: false to switch it off, or dryRun: true to watch without writing.
• Never touches a session written in the last 2 minutes (liveWindowMs).
• Verified before it writes. The pruned log has to satisfy the loader contract, or the original file is left exactly as it was.
• Destructive by design. The fragment rows are deleted. .orig is the only backup, and it is never removed for you. Read Destructive by design before enabling it.

Why not just hide the chunks? Folding them out of the session.history response is a real fix for the client, and this plugin does not replace it — the two run together. But the OOM does not happen in the client. It happens when DSH loads the log: the file is read, every row is parsed, and the chunk rows expand into millions of objects. If the file on disk is still 272 MB, skipping them in the response does not stop the next load from rebuilding them. The only way to stop that is to make the file smaller — see How this differs from the runtime approaches.

---

Why

A session log is a concatenation of zstd frames holding JSONL rows. Every token a model streams is written twice: once as a raw delta (assistant/chunk, and the packed rows text-chunks / reasoning-chunks / tool-call-chunks) and once inside the finalised assistant/message. Nothing reads the fragments. Replay does not read them, the model is never fed them — the projection layer does not even know the word chunks. They accumulate because they are write-only.

Measured on the session that motivated this plugin, a 1,813-turn conversation:

The file is small enough to sit on disk unnoticed and large enough to kill the process that loads it. And because every load pays that cost, the session gets slower and eventually unopenable — while the rows that cause it are provably unused.

After a cleanup, on the very same session:

⚠️ The "generation speed" row is an observation, not a promise, and it is not explained. ✗ What was observed: after the fragments were removed and load memory dropped, the generation speed reported by the UI recovered. No controlled before/after tok/s reading was taken, so this records the symptom only. Do not read a mechanism into it: context size, cache hit rate, time of day and the reasoning-effort setting were each suspected and each ruled out by data, and this is the only thing that was actually confirmed.

What the file size costs on the read path (measured 2026-10-15)

Opening or refreshing a conversation pays for reading its log: the multi-frame zstd stream is decompressed and every row is parsed. That is the server-side cost a page refresh sits on, and it scales with the fragment rows this plugin exists to remove — so it was measured directly.

60 logs that still had their pre-cleanup .orig beside them, i.e. the same conversation on both sides, decompress + JSON.parse of every row, median of 3 runs each:

Scope, so the number is not over-read ✗: this measures the log read (decompress + parse every row). It does not cover projection computation or client-side rendering — neither was measured. The ×0.72 case is a log whose pre-cleanup copy is barely larger than the current one, so it is noise on a small number, not a regression. Eight further pairs were discarded outright because their .orig predates later activity, which would have compared two different conversation lengths.

What it does

startDelayMs after boot, once:

1. Scan. Walk $DSH_HOME/sessions for session.jsonl.zstd, sort by size, biggest first. A file modified inside liveWindowMs is skipped-live — the plugin never swaps a log out from under a session that may be writing to it.
2. Threshold. Count fragment rows and their share of all event rows. Below minChunkRows or below minRatio, the file is below-threshold and is not touched at all. A few hundred kilobytes of fragments is not worth rewriting a log for.
3. Prune. Delete the fragment rows, then renumber seq and remap every cross-event reference onto the new numbering — sourceEventSeqs, surfaceOp.start/end, shadowedSeqs and data.shadowedRange. Events that the remap itself invalidates (a surfaceOp whose endpoints were dropped, a surface event left without a marker) are dropped too, in a fixpoint loop, so the result is consistent rather than almost-consistent.
4. Self-verify. The pruned log must satisfy the same contract DSH loads by: contiguous seq from 0, a marker on every surface event, no dangling references and no reference to a later event. Any violation ⇒ verify-failed ⇒ nothing is written and the log keeps its original size. Refusing to shrink a file is always acceptable; writing a file that DSH cannot load is not.
5. Replace atomically. Copy the original to <log>.orig (only if it does not already exist), write the new bytes to <log>.vacuum-tmp, then rename it over the log. A crash mid-write leaves the original intact.

The plugin cannot shrink an already-loaded session — it does not need to. Pruning the file means every future load is small, which is where the memory, the stutter and the OOM came from.

Running it on demand

The boot run has one practical flaw: it skips anything written inside liveWindowMs (2 minutes by default), which is usually the session you actually care about. And until this interface existed, the run reported only how many files it handled, so a scan that skipped 56 of 58 logs looked identical to a plugin that does nothing.

Two things follow from that, and both are now in place.

The log says why every file was skipped

run-end carries a count per reason, next to the handled it always had:

{"mark":"run-end","scanned":58,"handled":2,"vacuumed":2,"wouldVacuum":0,
 "skippedLive":56,"belowThreshold":0,"unreadable":0,"other":0}

handled keeps exactly the meaning it always had — the files this run actually acted on. The rest are the ones it did not act on, broken down by the same status strings the file lines use. Nothing about what gets cleaned changed; this is observability only.

clean_session_logs — ask for a pass now

An agent can call the Host tool clean_session_logs. It takes no required arguments and one optional one:

It runs the same guarded pass as the boot timer — the same liveWindowMs window, the same two thresholds, the same vacuumOne() that backs up to .orig, verifies, and replaces atomically. The on-demand entry point is not a bigger hammer; it just does not have to wait for the next boot.

It returns plain text, not JSON, so what you see in the transcript is what happened:

session-vacuum: DRY RUN - nothing was written.
Scanned 4 session log(s) under /home/you/.dsh/sessions.
Would clean: 2 log(s).
  - /home/you/.dsh/sessions/third/session.jsonl.zstd: 0.01 MB -> 0 MB (saves 0.01 MB), 120 fragment row(s) dropped, 92 event(s) kept
  - /home/you/.dsh/sessions/stale/session.jsonl.zstd: 0.01 MB -> 0 MB (saves 0.01 MB), 120 fragment row(s) dropped, 38 event(s) kept
Total that a real pass would save: 0.02 MB.
Skipped: 2 log(s), by reason:
  - skippedLive: 1 - modified within the last 10 minute(s), so DSH may still be writing it.
  - belowThreshold: 1 - under minChunkRows (20) or minRatio (0.2).
  - /home/you/.dsh/sessions/live/session.jsonl.zstd: SKIPPED - modified inside liveWindowMs, so DSH may be writing it right now
  - /home/you/.dsh/sessions/below/session.jsonl.zstd: SKIPPED - not enough streaming fragments to be worth a rewrite (8 fragment row(s), ratio 0.571)
run-end counters (logs/session-vacuum.jsonl): scanned=4, handled=2, vacuumed=0, wouldVacuum=2, skippedLive=1, belowThreshold=1, unreadable=0, other=0
NOTE: the session you are running in cannot be cleaned from inside a running DSH.
  DSH holds that session in memory, and rewriting its log underneath it can corrupt it.
  The liveWindowMs guard (10 min) already refuses it, so this pass cannot touch it - that is the only protection, and it was not weakened.
  What to do instead: stop DSH and run E:\Dsh百宝箱\清理会话日志.cmd,
  or leave it idle for at least liveWindowMs (10 min) and let a later boot clean it.

The session you are running in cannot be cleaned from inside a running DSH, and the report says so. DSH holds an active session in memory and keeps appending to its log; rewriting that file underneath it can corrupt it. The liveWindowMs guard already refuses it, and it was not weakened for this interface. To clean that particular session: stop DSH and run E:\Dsh百宝箱\清理会话日志.cmd, or leave it idle for at least liveWindowMs and let a later boot do it.

The web route (preview only)

A script can also ask, over HTTP:

POST /api/local/session-vacuum/run

It is POST-only and preview-only: any body that asks for a real pass is refused with 403, and a profile with dryRun: false refuses outright rather than exposing a destructive endpoint. A real pass belongs to the tool, where an agent owns the call. An empty body (or {} / {"dryRun":true}) is a dry run and the response body is the same text report.

Both entry points are inert while enabled is false, and neither one can bypass the guard.

How this differs from the runtime approaches

The same dead weight has a second, complementary fix: trim it at read time instead of at rest. dsh-host-history-lite intercepts the session.history / subagent.history RPC routes and folds finalised chunk deltas back into the final assistant/message before the response leaves the server. Both tools attack the same rows; they sit at different layers, and they can be used together.

Pick by what hurts: a sessions/ directory that keeps growing and a load that expands into gigabytes → this plugin. A history panel that takes seconds to open → the runtime approach.

A note on "nothing reads them". That is true on the server: the fragments are not part of the surface the model is sent, and no server-side projection reads them back. The client does replay them when it opens a finished conversation — which is exactly why the runtime approaches can drop them without changing what a finished conversation looks like.

When it runs

It only works on session logs that are not currently in use, so collecting a specific session is a sequence, not a switch:

1. Switch away from that session — don't just close DSH.
2. Leave it idle for more than 2 minutes (liveWindowMs).
3. Restart DSH.
4. About 20 seconds after startup (startDelayMs) the plugin sweeps and prunes it.
5. Next time you open that session it loads smaller, faster, and without the OOM.

A session you are using right now is never touched, and a log below the thresholds is never touched either.

It knows which sessions are in use (2026-10-18)

"Not modified for two minutes" does not mean "nothing has it open". A session the GUI keeps open but idle is exactly that shape, and rewriting it desyncs DSH: the file was renumbered 0..72555 while DSH still held its in-memory counter at 713504, so the next append wrote 713504 into a file numbered from zero. One log, two numbering spaces, a corrupted session.

The fix is not a ritual for you to perform — it is a fact the plugin can establish by itself. It listens for agent/created / agent/session-start and remembers every session this DSH process has loaded. Those are the dangerous ones, and they are skipped (skipped-loaded). Anything this process has never loaded is safe to rewrite, because DSH will read whatever file it finds the next time that session is opened.

So: install it, forget it. It keeps sweeping and it keeps your own working sessions alone. liveWindowMs remains as a second guard for sessions touched in the last couple of minutes.

The one moment a restart can clean is now used (2026-10-18)

The sweep used to start startDelayMs (20 s) after boot, by which time the GUI has usually reopened the session you were last using — so the session a user most wants cleaned was the one always skipped. It now sweeps immediately, from apply(), before anything connects: at that instant no session is loaded, so a recently-modified file only means "this was the last write before shutdown", not "someone is using it". The delayed pass stays as a second sweep.

Measured in the sandbox: beforeMB 85.56 -> afterMB 16.81, chunks 167029, kept 57999, .orig kept. That is the restart doing the cleanup, with no CLI and no waiting.

If a build cannot dispatch those agent events, the plugin falls back to the strict rule — inside a live DSH it writes nothing at all (skipped-in-process). And if you want to clean a session that is currently open, do it while DSH is stopped:

node lib/clean-cli.mjs --dry-run       :: report what would be rewritten
node lib/clean-cli.mjs                 :: do it (DSH must be stopped)
node lib/clean-cli.mjs --home C:\Users\you\.dsh

That entry point is the one path that can always write: it states "no DSH is running" by setting preBoot. allowInProcessWrite: true disables all of this for anyone who wants the old behaviour knowingly.

Install

One command, straight from the repository:

dsh plugin add --profile web github:mili1360/dsh-session-vacuum

That resolves the package over git, adds it to the profile's dependencies, and registers the bundle in dsh.profile.bundles on its own — verified on 0.2.0-rc.2. Then restart DSH: a registered bundle is only picked up at boot.

Works the same way from a local clone, if you would rather build from source:

git clone https://github.com/mili1360/dsh-session-vacuum.git
dsh plugin add --profile web ./dsh-session-vacuum

Doing it by hand instead

This is a plugin for a DSH profile. Add it to the profile — the same layer every other DSH plugin uses.

1. Make the package resolvable by your profile, then add it to the profile manifest ~/.dsh/profiles/<profile>/package.json:

   {
     "dependencies": { "dsh-session-vacuum": "link:/path/to/dsh-session-vacuum" },
     "dsh": { "profile": { "bundles": ["@deepseek-ai/dsh-base", "dsh-session-vacuum"] } }
   }
   

   Or, using the explicit patch layer, append to ~/.dsh/profiles/<profile>/cordis.patch.yml:

   - insert:
       - id: session-vacuum
         name: 'file:///path/to/dsh-session-vacuum/lib/index.js'
   

2. Restart DSH. Plugin rows load at boot.

3. It is already on. The shipped defaults are enabled: true with dryRun: false: it boots, waits startDelayMs, and collects logs that are past their thresholds. To watch it without letting it write, create ~/.dsh/dsh-session-vacuum/config.json:

   { "dryRun": true }
   

   enabled is read once at boot, so changing it needs a restart. dryRun: true then logs what it would remove without writing anything: watch ~/.dsh/logs/session-vacuum.jsonl until the numbers look right, then set "dryRun": false.

To remove it, delete the row you added and restart. Nothing else is left behind — the config file, the log file and any .orig backups are the only things this plugin creates.

Node 22.15 or newer is required: the plugin uses zlib.zstdDecompressSync. On an older runtime it does nothing at all rather than doing something wrong — the decompression call fails, no rows are counted, and every file reports below-threshold.

Requirements / compatibility

After installing, confirm it actually works — a profile that boots is not proof. A plugin can load cleanly, serve its routes and still break the host; we have seen exactly that. So the acceptance test is: create a new session and send it one message, then check $DSH_HOME/logs/session-vacuum.jsonl for a run-start / run-end pair about startDelayMs after boot. If new sessions cannot be created at all, remove the mount line above and restart.

Developed and tested against DSH 0.1.0-rc.7 — a customised build, not a stock upstream release. Another version may register its services differently. This plugin declares inject: ['tools'] and uses ctx.effect, ctx.tools.register (the on-demand tool), and optionally ctx.webServer.register (the preview route).

Newer DSH: the v4 session format — supported, with two honest gaps

DSH 0.2.0-rc.2 writes session logs as session.v4.jsonl.zstd (format v4), and v4 has no fragment rows — the packed delta runs live inside assistant/message's data.stream. The row-level pruning that works on v0/v1 therefore matched nothing there, and the plugin would have loaded cleanly and done nothing at all. That path is now implemented (it compacts the runs inside the event while the loader's own reader verifies the result), and it was verified end-to-end against real sessions in a sandbox: a 5-turn session went 27,715 → 17,071 B, and the pruned log still loaded, with every message byte-identical after the fold.

What is not verified, and you should weigh before relying on it:

• assistant/attempt has no real-data coverage. The code path is exercised and the verifier refuses any non-equivalent result, but this build only writes that event for a failed, retried, cancelled or stream-error attempt — all of the test sessions succeeded, so no real sample exists. Treat it as unit-level assurance, not measured.
• The v4 path has no automated regression. The bundled test suite covers the v0/v1 row format only; v4 is currently held up by the sandbox end-to-end runs described above.

Both gaps are stated so that a silent failure — the exact thing this plugin exists to remove — is not hidden behind a green test run.

Why inject is not optional. Cordis injects services on demand: a service that is not declared in inject does not exist as a property on ctx, and reading it throws cannot get property "tools" without inject. That is not theory — the first build of this interface was deactivated by exactly that line on a real boot. webServer is deliberately not declared, because a headless profile has no web server: the route uses the optional ctx.inject(['webServer'], ...) form instead, so a profile without one still gets the tool. If ctx.inject is unavailable too, the plugin logs mark: "route-failed" and carries on.

If your build does not match, the plugin deactivates and says so — it does not stop the profile from booting. No timer is armed, no session log is touched, and one line lands in logs/session-vacuum.jsonl:

{"mark":"deactivated","reason":"registration-failed","step":"services","error":"..."}

step names what failed: services (ctx.effect missing), ctx.effect (it threw), or ctx.tools (no usable tools registry). A failing route is not in that list: it is logged as mark: "route-failed" and the plugin stays active with its tool, because the route is a convenience and the tool is the point. @deepseek-ai/dsh-tools is resolved at plugin load to get defineTool; if it cannot be resolved the tool is still registered in the bare-object form (ctx.tools.register({...})) rather than disappearing.

If the profile will not start at all after installing, remove the mounted lines from <profile>/cordis.patch.yml (or drop "dsh-session-vacuum" from dsh.profile.bundles, if that is how you installed it) and restart:

- insert:
    - id: session-vacuum
      name: 'file:///path/to/dsh-session-vacuum/lib/index.js'

Configuration

$DSH_HOME/dsh-session-vacuum/config.json (falling back to ~/.dsh/…). Missing file, or a file that is not valid JSON, means