tokensqueezer

TokenSqueezer

A DeepSeek Harness plugin that reduces what an agent costs to run. It works on the model's own output — not on the input it is fed.

English | 中文

---

Read this first: what actually saves tokens

Two mechanisms with completely different economics live in this plugin, and conflating them is the easiest way to be disappointed.

Compression cannot make the current answer cheaper. That money is spent. What it does is shorten the transcript, so every later request in the turn carries fewer bytes — a saving that compounds and is invisible at the moment it is made.

The plugin never touches usage, so reported token counts stay truthful.

---

Install

# 1. Put this repository somewhere permanent, then link it into a profile.
#    The profile manages its own node_modules; a link: install keeps the plugin editable.
cd <your DSH profile directory>
pnpm add link:<path-to-this-repository>

# 2. Enable it. The bundle ships cordis.patch.yml, which inserts the row for you.
#    In <DSH_HOME>\profiles\<profile>\cordis.patch.yml:
- id: tokensqueezer
  disabled: false

Restart the application. The two halves reload by different mechanisms, and the asymmetry is worth knowing before it wastes an afternoon:

So a UI that updates while behaviour does not is the normal symptom of a stale Host. status prints the file the Host was loaded from, which makes the question a comparison rather than a guess:

Host code: <path>\index.js @ 2026-09-30T04:26:12.732Z.

---

How it works

Host half — five surfaces, each installed independently

Each is its own installation step, so a failure names itself in status (Installed: …) and does not take the other four down with it.

Client half — two seats

• conversation.composer.dock — a readout below the composer. It polls the status route every 3s and shows the session's saving. When no number is available it says so instead of showing a placeholder.
• settings.section — a settings page built from the shipped primitives (Tag, Switch, SegmentedControl, Input, CodeBlock, Button), grouped by what each setting actually saves, localized through the shipped locale service.

The pipeline

lib/engine/output.js splits a payload into fenced-code and prose segments first, so no code block ever reaches the semantic layer. Everything else flows through:

1. L1 lossless — ANSI and control removal, carriage-return redraws, trailing whitespace, blank-line runs, repeated line/block folding.
2. L2 normalisation — path relativisation against the session cwd, optional symbol interning. Both skipped for executable kinds.
3. L3 semantic — one compressor per detected kind: json, yaml, diff, stacktrace, log, terminal, testreport, filetree, searchresults, deptree, markdown, html, code, prose, plus answer and reasoning, which fold at answer scale.

Reversibility

Everything the lossy layers remove is interned into a content-addressed store (8 hex characters of sha1) and replaced by a bare ⟦TS:id⟧ marker, with a legend line at the end of the payload. The same text always yields the same id, so a second pass over compressed output is a no-op. The model retrieves originals through the tokensqueezer tool's expand action.

---

The code guarantee

A tool result that is file content or a patch is not compressed semantically. This is enforced in three places, and the reason there are three is a mistake worth recording:

The first implementation called engine.compress(text, { lossy: false }) and assumed that was enough. Measured on a 40-line repeated body, it returned three lines. The repeat-line folding that damages code runs in the lossless stage, so switching the lossy layer off happens upstream of the damage.

So:

1. The tool name decides the kind. toolContentKind() pins read/read_file/view/cat/write/ edit/str_replace → code, and diff/patch → diff. Heuristics are the wrong instrument here: a source file that detectKind happens to read as log would have its repeated lines folded.
2. Those kinds skip the pipeline entirely and receive losslessCleanup only. EXECUTABLE_KINDS is not the guard — it only protects L2, while compressCode/compressDiff do fold code bodies.
3. The result must be net-shorter. A payload whose final text — marker and legend included — is not shorter is discarded unchanged. This is what makes "compress everything" safe rather than self-defeating, because a 13-character marker plus a ~100-character legend would otherwise grow small payloads.

Measured on the shipped engine:

read_file, 40 repeated code lines   1079 → 1079   40/40 lines survive
patch, 30 added lines               1231 → 1231   30/30 lines survive
code with ANSI escapes and blanks    1122 → 1111   41/41 lines survive, escapes stripped

---

Configuration

Every key has a default; nothing needs configuring. The floors default to 0, so an ordinary payload always enters the compressor — and the net-shorter rule above decides whether the result is kept.

outputTokenCap and reasoningEffort apply to every agent, subagents included — deliberately, so a delegation cannot quietly outspend the budget it was given.

---

Tooling

The plugin ships its own verification, because the engine layer was never the part that broke:

$node = '<node executable>'
& $node scripts\linkcheck.mjs      # every relative import resolves to a real export
& $node scripts\selftest.mjs       # engine, 9 sample kinds: 122537 → 32812 chars (73.2%)
& $node scripts\hosttest.mjs       # Host half, fake ctx and a session as strict as the real one
& $node scripts\clienttest.mjs     # Client half, fake loader, a real render, and the schema the settings form needs
& $node scripts\reasontest.mjs     # skeleton folding: skeleton kept, references reversible
& $node scripts\zstdtest.mjs       # session-log reader: concatenated frames, truncated tails
& $node scripts\archtest.mjs       # architecture digest

Three diagnostics exist for the questions the UI cannot answer:

hosttest.mjs uses a fake session that enforces the real rules — an assistant/message replacement must not carry sourceEventSeqs, must carry turn/step while that step is open, and surfaceOp must be exactly {op, startSeq, endSeq}.

---

Platform constraints

These were all found the hard way, and each one cost at least one restart. They are properties of the Harness, not of this plugin.

• A profile-installed bundle cannot import @deepseek-ai/*. The packages live inside the application archive; Node's resolution never reaches it, and the entry fails with a bare failed to import. So the Host half is a functional plugin with no framework imports, and the two helpers it needs are vendored.

• cordis resolves a service on a context only when that context injects it. An undeclared service is not merely unavailable — reading it as a property throws. That throw is what silently killed one installation step here. inject is a gate, and both halves declare what they need (['tools', 'webServer'] and ['slots', 'locale', 'configForms']).

• A plugin Config is two different protocols. cordis activation validates through Standard Schema (Config['~standard'].validate, synchronous); DSH's settings projection wants a schemastery brand (Symbol.for('schemastery'), type, meta). Supplying only the brand satisfies the inspect provider — which reports a healthy status: "schema" — while the entry goes to fiberPhase: "failed". An inspect provider is a read-only projection, not a validator.

• A client-visible projection needs compile-time declaration merging into SessionProjectionMap plus a wire codec, and a typert Remote namespace needs generated code. Neither is available to a plain-JS plugin, which is why the UI reads a read-only HTTP route instead.

• SessionReference documents behaviour, not a content API. It has paging, minima and lifetime semantics, but no public way to read the loaded messages.

• agent/request, llm/stream and tools/post-execute are waterfalls and can legitimately replace what they return. agent/assistant-stream is emit only and can never be rewritten.

---

Honest limitations

• Short content does not compress, and should not. With every floor at zero the compressor is still reached; it simply finds nothing to remove. A 46-character answer has no foldable middle, and an 11-character tool result has no redundancy. Pushing further would be deleting information, not saving it.

• Measured savings on real sessions are modest — single-digit to low-double-digit percent of the blocks that qualify, with roughly half of all blocks having no redundancy at all. The large numbers are in the prevention knobs, and two of those (outputTokenCap, reasoningEffort) default to off because they change what the model produces.

• The settings page's write path is verified against the contract, not against a browser. The controls call mutate(ops, revision) — read from the shipped SettingsFormModel, not guessed. Two gates decide whether a hand-built schema is served at all, and lib/config.js now satisfies both: the entry's Config must answer "toJSON" in schema, and at least one field must carry meta.volatile. Miss the first and describe() drops the row before it reads anything else; miss the second and volatileForm() drops it one step later. Either way no namespace appears, configForms.get() finds nothing, and every switch and number input renders disabled. clienttest.mjs pins both gates, so the page cannot silently go grey again. What is still unexercised is the round trip through a live browser: change a value, watch for Saved, and confirm the Host applies it.

• toolSkip is not editable from the settings page. It is the only non-scalar setting, and an array node that failed to rehydrate would drop the page back to status: loading — greying every control, the exact failure above. It stays editable through the tokensqueezer tool's set action and the profile patch row.

• The dock readout depends on the status route. If the route is not reachable it says so; it never renders a number it does not have.

• The reference store is per session and bounded. expand resolves within the session that produced a marker, and the LRU keeps 32 sessions warm. Originals always remain in the durable log.

---

Layout

index.js            Host half: five surfaces, the tool, the prompt sections, the status route
client.js           Client half: the composer dock and the settings page
cordis.patch.yml    bundle patch: inserts the tokensqueezer row
package.json        bundle and client declarations
lib/config.js       the plugin Config (Standard Schema + schemastery brand)
lib/session-message.js  the v4 producer-source rule still relied on
lib/arch.js         architecture digest
lib/engine/
  compress.js       orchestrator
  output.js         fenced-code split, code protection, answer-scale wiring
  detect.js         content-type detection
  textsafe.js       L1 lossless
  refs.js           content-addressed reference store
  dedupe.js         line and block folding
  normalize.js      path and symbol normalisation
  tokens.js         token estimation
  kinds/            one compressor per content kind
locale/             i18n: en.json, zh.json
scripts/            the seven suites, the shared session-log reader, and the three diagnostics

License

MIT.