dsh-blackhole

dsh-blackhole

English | 简体中文

A DSH port of k0valik/pi-blackhole's brief compiler (VCC conversation-compiler), wrapped in a dual-pipeline compaction engine.

Drop-in replacement for the official @deepseek-ai/dsh-compaction-basic backend that routes inside the single summarize() hook:

• OpenAI (or any provider with a configured Responses endpoint) — the

official remote path: summarization is delegated to the model server via the Responses API compaction_trigger (the same protocol Codex's remote compaction uses), and the opaque replacement history is persisted per session;

• remote failure (no endpoint, missing key, timeout, protocol error) —

by default the region is compiled locally by a deterministic VCC-style compiler (zero model calls, near-lossless (seq N) pointers back into the durable session log), with the native one-shot LLM summarizer available via fallbackMode: "llm";

• every other provider — the local compiler, straight away, no network.

Remote compaction branch ports (MIT). The local compiler is a port of pi-blackhole's `brief.ts` compiler (VCC conversation-compiler, MIT; rule lineage: lllyasviel/VCC → dsh-compaction-instant) — the differential suite pins content-level equivalence against the real pi-blackhole sources (see *Equivalence* below).

Why

· pi · DSH (this plugin) Remote compaction v2 · via algal/kky42/lll9p extensions · remote branch of the routing engine Zero-cost local compaction · pi-vcc / pi-blackhole (algorithmic) · local compiler branch (fallbackMode: "instant") Compaction seam · extension event hooks · CompactionEngine.summarize() — the official single customization hook Fallback chain · — · instant → (optional) native LLM → fail Opaque window persistence · pi session JSONL compaction.details.remoteCompaction · per-session JSON state under $DSH_HOME/plugins/dsh-blackhole/state

How it works

1. DSH decides to compact (pressure / overflow / /compact) and calls the

engine's summarize() hook with the replayed conversation region.

1. The routed provider/model is resolved (durable request header first, agent

options second) and matched against the configured endpoints.

1. Remote branch (matched endpoint): the region is converted into

Responses input items (text, reasoning, tool calls/results), a compaction_trigger is appended, and the compaction response streams from {baseUrl}/responses. The assistant-authored summary becomes the readable checkpoint; the opaque compaction item plus the retained tail of user messages (Codex-style 20K token budget) is persisted per session. A later remote compaction for the same provider/model replays that opaque window and appends only the new trailing surface.

1. Fallback: any remote failure logs a warning and hands the region to

the local compiler (instant, default) or the native LLM summarizer (llm). instant-then-llm tries the compiler first and only then the LLM.

1. Local compiler branch (no endpoint match, or fallback): a deterministic

zero-model pass over the region that keeps ONLY original tokens — every tool call becomes one line * bash "ls" (seq 2 -> result 3), tool results never occupy entries (one recall away via the pointer), reasoning is elided, long text truncates with ...(truncated from seq N), noise XML is stripped, and a token cap drops the oldest rows first with an explicit [N entries elided: seqs a-b] note. The (seq N) pointers resolve against the REAL durable session log (each flattened message is matched to its stored event by id), so a recall tool mounted next to this engine restores the exact original content.

The engine subclasses the official BasicCompactionEngine, so all pressure / retention / policy knobs, the token-meter pricing, and the context-overflow recovery behave exactly as upstream.

Recall and file recovery

The bundle mounts a dedicated tools unit on both the host and standard-preset planes. It waits for the tools service and registers recall through the official defineTool() adapter; the compaction engine does not register tools. @deepseek-ai/dsh-tools is a required peer supplied by DSH. The optional /blackhole-recall command still uses the host's dsh-commands service. Both surfaces read DSH's lossless in-memory event log and the original tool-call arguments, so they continue to work after a local or remote compaction checkpoint without depending on pi's JSONL files.

Typical tool input:

{
  "query": "SecretNeedle",
  "mode": "hybrid",
  "scope": "lineage",
  "page": 1,
  "limit": 5
}

The same search is available interactively as:

/blackhole-recall SecretNeedle mode:hybrid scope:lineage page:1 limit:5

Supported recall forms are:

Form · Result text (parser) or explicit regex (/parser/i, /hook\ · inject/) · ranked transcript search; hybrid mode also searches bounded Write/Edit content #N · expand exactly durable event N, with complete prose/tool results up to the 50 KB response cap; file bodies remain referenced by path #N:path or #N:file · preview one file operation, or auto-select it when the entry has one file #N:path:full · return the complete operation body up to the 50 KB display cap #N:path:offset / #N:path:offset:limit · zero-based line paging (limit defaults to 30) mode:file · search only entries with content-bearing file operations mode:touched · list paths from content-bearing Write/Edit-style operations and their durable sequence numbers (5 paths per page by default) scope:all · include events outside the active lineage (the default is the active lineage) expand: [N, ...] · force up to 50 specific entries into a result page even when the query does not match

Write/Edit extraction recognizes content, edits, oldText, and newText along with common path keys. Ordinary ranked pages clip prose blocks to about 300 characters, tool results to about 200 characters, and call arguments to 300. Queries use keyword search unless explicitly wrapped as /pattern/flags; punctuation in model names and paths does not enable regex. Existing bare regex queries must add delimiters, for example hook|inject → /hook|inject/. Transcript hits outside the head preview include up to three bounded verbatim excerpts from the redacted search corpus. These are partial context; use #N or expand: [N] before relying on the complete message. File-body match notes remain separate and full Write/Edit bodies still require file drill-down. Write/Edit transcript arguments retain string paths and scalar metadata; both standard body fields and unknown string/object payloads are redacted, including mixed provider schemas. Known file bodies remain indexed separately. #N restores the complete prose/result blocks and 2,000 characters of each call's arguments — the preview's 300-character cap truncated 222 of 1,004 command lines in the readback, which defeated the point of drilling into an address. File bodies are available only through #N:path. Search indexing clips each file operation to 10 KB; every recall response, including exact expansion and offset paging, is capped at 50 KB. Search/touched limit values are capped at 100 entries and explicit expansion at 50 entries. Unsafe, oversized, or backtracking-prone regular expressions fall back to ordinary token search. Ambiguous paths are reported with candidate paths instead of being silently selected. A 12-character lowercase hexadecimal query is reserved for observational-memory IDs and returns an explicit integration message.

This is intentionally close to pi-blackhole's transcript/file recall, but it is not OM-complete yet: the OM observation/reflection ledger, OM-ID lookup, and automatic reverse recall from OM records are a TODO. VCC-style summary output and recall are therefore coupled through durable DSH sequence numbers, not through OM memory IDs. scope:all always stays within the current DSH session and never loads sibling fork/session logs. Stock DSH sessions do not currently expose the optional lineageSeqs projection, so lineage and all differ only when a host supplies that projection.

Install

Requires DSH 0.1.0-rc.6 (web or headless profile).

# into the profile that serves your sessions
dsh plugin --profile web add dsh-blackhole
# or from a local tarball:
npm pack dsh-blackhole && dsh plugin --profile web add ./dsh-blackhole-0.2.0.tgz

The bundle patch:

• disables the official compaction-basic row (already disabled by the

web-app bundle — idempotent),

• re-enables command-compact (the /compact command, which the web-app

bundle disables; it is backend-independent),

• inserts this engine under its own row.

Exactly one compaction service ends up mounted; the dual pipeline lives inside the engine.

Configure

In the profile's settings (or ~/.dsh/settings.yaml under the plugin key):

dsh-blackhole:
  # inherited BasicCompactionEngine knobs (all optional, same defaults as
  # @deepseek-ai/dsh-compaction-basic):
  thresholdRatio: 0.75
  retainRatio: 0.25
  auto: true

  # Responses-assisted knobs:
  endpoints:
    # direct OpenAI Responses
    - provider: openai
      baseUrl: https://api.openai.com/v1
      apiKeyEnv: OPENAI_API_KEY
    # any OpenAI-compatible gateway that forwards the Responses API
    - provider: '*'
      baseUrl: https://my-gateway.example.com/v1
      apiKeyEnv: MY_GATEWAY_KEY
    # official Codex backend (needs the beta-feature header)
    - provider: openai-codex
      baseUrl: https://chatgpt.com/backend-api
      apiKeyEnv: CODEX_TOKEN
      headers:
        OpenAI-Beta: responses=experimental
        x-codex-beta-features: remote_compaction_v2
  timeoutMs: 120000      # abort the remote attempt after 2 min
  fallbackMode: instant  # instant | instant-then-llm | llm
  format: blackhole      # blackhole | ledger | index (see "Checkpoint formats")
  retainedMessageTokenBudget: 20000
  persistRemoteHistory: true
  stateDir: ~/.dsh/plugins/dsh-blackhole/state

  # Local compiler knobs (all optional):
  compileMaxTokens: 8192     # floor; scaled by checkpointScale on big regions
  checkpointScale: 0.1       # cap = max(compileMaxTokens, shadowed x scale)
  checkpointCap: 65536       # hard ceiling

  # ledger format knobs (all optional; defaults are calibrated to this
  # plugin's tokenizer, which runs ~1.094x cl100k):
  ledgerBudget: 656          # fact section (P/F/C), partitioned F35 / P45 / C20
  ledgerTimelineBudget: 438  # write timeline (W rows)
  ledgerAtomChars: 150       # per-atom character cap
  textTokens: 512            # per assistant text/reasoning block
  userTextTokens: 1024       # per user text block
  toolCallTokens: 128        # per tool one-liner
  includeReasoning: false
  stripNoiseXml: true
  noisePatterns: []          # empty = built-in VCC patterns
  toolArgTools: []           # empty = built-in whitelist
  hideTools: []              # e.g. [TodoWrite, ToolSearch]

  # Index pool sizes are minimums; automatic sizing includes carried state.
  # The overall compileMaxTokens/checkpointScale/checkpointCap limit still applies.
  indexBudget: 2048          # one `#seq tool@call arg` row per tool result
  diagnosticsBudget: 768     # F section: verbatim failure lines
  requirementBudget: 256     # U section: the user's own sentences
  requirementCount: 8        # optional goals only; 0 disables goals, active rules stay whole
  requirementChars: 160     # optional goal character cap, not an active-rule cap

Endpoint matching: provider matches the DSH provider route the latest request ran on (exact, or * for any); model matches exact id or a * glob. Authorization uses apiKeyEnv (env var name) or apiKey (inline — never commit a real key).

fallbackMode semantics (applies to remote failures only):

mode · remote ok · remote fails instant (default) · server summary · local compiler (zero model calls) instant-then-llm · server summary · local compiler; LLM only if the compiler itself fails llm · server summary · native one-shot LLM summarizer

The legacy fallbackToLlm boolean still maps: false forces instant; true behaves like the default. Targets with no matching endpoint always take the local compiler (or, in llm mode, the native LLM).

Runtime switching: /blackhole-mode

The bundle patch mounts two dedicated plugin units inside the preset compaction group, mirroring the official shapes (dsh-command-compact / dsh-tool-todo):

• dsh-blackhole/commands — inject = ['commands', 'compaction'] — registers

/blackhole-mode and /blackhole-recall;

• dsh-blackhole/tools — inject = ['tools'] — registers the recall tool

through the official defineTool() wrapper (enforced JSON-Schema arguments, structured output render).

They appear wherever /compact and the built-in tools do — no config edit, no restart to take effect:

/blackhole-mode                 # status: current chain, remote gate, format, endpoints
/blackhole-mode instant         # switch the fallback chain (instant | instant-then-llm | llm)
/blackhole-mode llm             # e.g. hand this session's checkpoints to the LLM summarizer
/blackhole-mode remote off      # close the remote gate; every target uses the local chain
/blackhole-mode remote on       # reopen it
/blackhole-mode reset           # drop all runtime overrides, back to config defaults

Switches are sticky: each one persists into runtime-mode.json inside the plugin state dir (default $DSH_HOME/plugins/dsh-blackhole/state) and is re-applied over the config defaults on every DSH start — so an override made yesterday survives today's restart. The plugin config (fallbackMode / endpoints) remains the durable source of defaults; the override file only carries runtime deltas, and the plugin never writes the host's profile YAML. /blackhole-mode reset (or deleting the file) returns everything to the config defaults. Disabling remote — or any local/LLM checkpoint landing — removes the session's persisted remote replacement history, keeping the "local checkpoint supersedes the remote window" invariant intact.

Checkpoint formats: blackhole, ledger and index

The format is a second, orthogonal axis: it decides what a checkpoint *keeps*, not where it comes from. Both formats run on the same engine, and the ledger path makes no model call at all — it short-circuits both the remote gate and the fallback chain.

· blackhole (default) · ledger · index body · VCC-style entries: prose, (seq N) read-back pointers · P paths / F failure lines / C commands / U the user's own instructions, verbatim; plus a W seq tool path write timeline · one #seq tool@call arg row per tool result, an F diagnostics section, a U instructions section prose · kept (compressed) · none · none paths kept · 32.4% · 95.8% (@1200t) / 91.3% (@600t) · not kept — carried by the row's argument and the F lines failure lines kept · 12.6% · 100% · 100% results addressable · all, but only inside prose · 15/152 (write rows address calls, not results) · 152/152 ratio · ~2–5× · ~23–60× · ~62× answers "why was X rejected" · yes · no — use the recall tool · no — but every omitted result has an address

Those retention numbers are for 1–3k-token regions (the probe set they were

calibrated on). They do not transfer to regions an order of magnitude

bigger: on five real 97k–175k-token regions the default ledgerBudget 656

keeps 44.8% of paths and 42.1% of failure lines, reaching 95.9% / 65.8% only

at ledgerBudget 1968. Full numbers: artifacts/dsh-real-session-test.md.

Those figures are lower than the 49.0% / 53.4% this README used to carry — not

a regression but an honest denominator. The old extractor missed every typed

error (TypeError:, ReferenceError: — the ^ anchored only the first

alternation) and silently dropped failure lines over 200 characters, so a real

region counted 58 failures where there are 76. More failure lines compete

for the same F/P/C budget, which is why the path column moves too.

Three properties are deliberate and easy to get wrong:

1. The budget is partitioned (F 35% / P 45% / C 20%, remainder

backfilled), never filled in priority order. Order-filling starves failure lines — measured 81.8% vs 100% retention at the same 600-token budget, because paths always win the race.

1. Ledgers accumulate. A second compaction parses the superseded

checkpoint back into atoms and merges them, so paths recorded by the first compaction retain their source anchors. The old checkpoint is parsed as state, rather than extracted again as fresh evidence.

1. The user's instructions are quoted, not summarized (U). A ledger has