REMI

REMI

Unreleased framework

The unreleased framework adds explicit project scopes, cross-session Hebbian graphs, lifecycle deletion/scrubbing, a shared local manager and paired baseline/ablation validation. Hebbian and affect remain enabled in full mode; disabling them is an experimental control only. Offline tests do not establish live DSH compatibility or production cost savings. See the framework guide, evidence audit, and reference review.

Adaptive Memory for AI · DeepSeek Harness plugin · v0.3.0-alpha.1 development

English | 简体中文

REMI is a context and long-term memory plugin for DeepSeek Harness. It observes durable DSH session events, stores retrievable memories in SQLite, and assembles a bounded runtime-context window before each model request.

REMI is not a standalone web application, does not replace the Harness Agent Loop, and does not train or fine-tune a model.

Status: Experimental. Do not use REMI in production environments yet.

See CHANGELOG.md for release changes. v0.2.1 evidence remains in the validation note. The v0.3 current-truth path has completed one isolated live DSH run; protocol, evidence, and remaining caveats are recorded in the validation note.

Features

• Observes user/message, assistant/message, and tool/result through session/event.
• Stores memories, affect hints, retrieval traces, and association edges with the built-in Node.js node:sqlite module.
• Captures the current query at agent/inbox/claimed and injects one bounded runtime-context snapshot through system-prompt/assemble.
• Ranks candidates with hashed bag-of-tokens cosine, lexical Jaccard, recency, importance, association, and affect signals.
• Window hygiene removes the current-turn echo, questions, acknowledgements, irrelevant explicit sibling entities, duplicate statements, and non-assertive future language from ordinary factual windows. Conditional plans remain retrievable when the query explicitly asks for them.
• Content-aware ranking removes common recall/output boilerplate from a scoring-only view and uses corpus-IDF weighted lexical evidence. Original memory text is preserved; no model is trained.
• Reinforces bounded Hebbian edges between memories selected in the same retrieval window.
• Optionally requests structured affect JSON through DSH ctx.llm.stream(); no model training is involved, and failures fall back to a local heuristic.
• Extends the official BasicCompactionEngine, preserving the Harness compaction transaction while replacing only the checkpoint summarizer.
• File-loaded Desktop builds resolve the compaction engine from the running Desktop host, avoiding development-peer surface-format mismatches.
• Excludes plugin injections, skill catalogs, and agent instructions from memory and summaries. Historical injected records remain stored for audit but are quarantined from retrieval; clean official continuity checkpoints may still be re-summarized.
• Emits JSONL telemetry without raw query text and stores complete RetrievalTrace records in SQLite.
• Optionally recognizes narrow, explicit Chinese/English correction forms and atomically marks one uniquely matched active fact as superseded. Current queries exclude old versions; explicit history queries may include them.

Retrieval ranking controls

contentAwareRetrievalEnabled defaults to true; set it to false for the original raw-v0.2 scoring ablation. windowHygieneEnabled also defaults to true; disable only this option for the content-aware-v1 ablation without the new window admission rules. minLexicalCoverage defaults to 0.15 and prevents recency, importance, affect or hash collisions alone from admitting unrelated candidates. Questions and acknowledgements remain in SQLite and traces for audit, but do not enter a window-hygiene-v2 model window or Hebbian reinforcement. Classification is a bilingual heuristic, not verification that a claim is true.

Traces include scoringVersion, contentKind, utilityFactor, queryCoverage, focusOverlap, and an explicit admission decision such as current-query, conditional, focus-mismatch, or redundant. They contain scores and hashes, not the extracted focus terms. Content-aware embeddings are recomputed from the scoring view, so existing raw embeddings and records require no destructive migration. The benchmark includes a no_window_hygiene ablation; do not compare reports from different scoring versions without labeling them.

Requirements

• Node.js ^22.19.0 || >=24.0.0
• pnpm 11.x
• DeepSeek Harness API 0.1.5-rc.2, or a DSH Desktop build verified for compatibility

The compile-time baseline is DeepSeek Harness commit c291e7961a515f6d7af9304e7fd1d257929aef26. See API compatibility for the verified contracts and upgrade checklist.

Install and build

git clone https://github.com/Hoshino910/REMI.git
cd REMI
pnpm install
pnpm build
pnpm typecheck
pnpm test

The built DSH plugin entry is:

packages/dsh-adapter/dist/index.js

Add REMI to DeepSeek Harness

Two configuration examples are included:

• examples/deepseek-harness/cordis.yml is a replacement compaction group for current standard/ptc presets.
• examples/deepseek-harness/cordis.patch.yml is an overlay for profiles that still mount id: compaction-basic at the root.

Update the plugin and data paths before starting DSH:

- id: selective-memory
  name: 'file:///C:/path/to/REMI/packages/dsh-adapter/dist/index.js'
  config:
    databasePath: 'C:/path/to/dsh-data/remi.sqlite'
    telemetryPath: 'C:/path/to/dsh-data/remi-telemetry.jsonl'
    telemetryEnabled: true
    transparentWindowEnabled: true
    currentTruthEnabled: false
    retrievalTokenBudget: 1400
    retrievalLimit: 8
    minRetrievalScore: 0.12
    similarityWeight: 0.55
    recencyWeight: 0.15
    importanceWeight: 0.10
    associationWeight: 0.15
    emotionWeight: 0.05
    hebbianEnabled: true
    affectEnabled: true
    emotionAnalysisEnabled: false
    compactionSummaryTokenBudget: 900
    compactionThresholdRatio: 0.80
    compactionRetainRatio: 0.16
    autoCompaction: true

On Windows, the packaged DSH Desktop ESM loader requires a file:///C:/... URL for the plugin entry. A plain C:/.../index.js path is not accepted.

Start a profile with an additional patch:

dsh --profile web --patch C:/path/to/cordis.patch.yml

Only one ctx.compaction provider may exist in a Cordis realm. Disable the original compaction-basic provider or replace the complete isolated compaction group as shown in the examples.

Configuration reference

Storage and telemetry

The SQLite database contains raw conversation text. Protect it at least as strictly as the Harness session store. Do not commit SQLite, WAL, SHM, or telemetry files to Git.

Retrieval and transparent window

Weights are normalized at runtime. retrievalTokenBudget applies only to the rendered REMI <memory-context> block; it is not a hard limit for the complete model request.

Current-truth lifecycle (experimental)

This alpha feature deliberately accepts only explicit forms such as Change the Cedar service port from 6389 to 8247 or 将 Cedar 服务端口从 6389 改为 8247. The subject and optional previous value must identify exactly one active memory. If no target is found, several records match, or the message is conditional (may, might, 可能, 考虑), REMI stores the event normally and does not mutate any existing record.

Superseded records remain in SQLite for audit. They are excluded from ordinary retrieval and Hebbian reinforcement, but may be returned when the query explicitly asks for history. This parser is a conservative lifecycle control, not a general fact extraction or truth-verification model.

Hebbian associations

An edge means that two active statement memories were selected together. Graph expansion cannot bypass the lifecycle, statement-kind, lexical-coverage, or focus gates. It only reorders otherwise eligible candidates. An edge does not establish factual correctness, causality, or a user preference, and it does not train a neural network.

Affect analysis

Model analysis uses the provider already configured in DSH. It is an inference-only request with a strict JSON prompt. On failure, REMI records the error and continues with a local heuristic.

interface EmotionVector {
  valence: number      // -1..1
  arousal: number      // 0..1
  dominance: number    // 0..1
  labels: string[]     // up to three short labels
  confidence: number   // 0..1
  source: 'model' | 'heuristic' | 'fallback'
}

Affect values are low-weight retrieval hints. They must not be treated as a psychological diagnosis or as a durable fact about the user.

Compaction

REMI subclasses BasicCompactionEngine. Locks, token-pressure checks, surface replacement, manual flush, and the compaction/start → compaction/summary → compaction/end transaction remain owned by DeepSeek Harness.

Architecture

flowchart TD
    A["session/event"] --> B["SQLite memory store"]
    B --> C["hashed cosine + lexical Jaccard"]
    D["agent/inbox/claimed"] --> C
    C --> E["recency + importance + association + affect"]
    E --> F["token-budget packing"]
    F --> G["system-prompt/assemble"]
    G --> H["replaceable runtime-context snapshot"]
    I["DSH token pressure"] --> J["official compaction transaction"]
    J --> K["extractive continuity checkpoint"]

Event observation

The observer converts text from user, assistant, and tool-result events into MemoryRecord values. When current-truth handling is enabled, explicit user corrections first pass through the deterministic resolver; a unique match is replaced in one SQLite transaction. Messages generated by REMI and compaction checkpoints are ignored to prevent feedback loops. (session_id, source_event_seq) provides idempotency when an event is replayed.

REMI observes live events after installation. It does not backfill older session history automatically.

Retrieval scoring

The local deterministic similarity placeholder is:

Similarity = 0.65 × hashed bag-of-tokens cosine
           + 0.35 × lexical Jaccard

The default candidate score is:

Score = 0.55 × Similarity
      + 0.15 × Recency
      + 0.10 × Importance
      + 0.15 × Association
      + 0.05 × Affect

Recency uses 1 / (1 + ageDays / halfLifeDays). After packing candidates, REMI measures the complete rendered <memory-context> and removes the lowest-priority selected items until the configured budget is satisfied.

Transparent context window

REMI captures the current direct user query at agent/inbox/claimed, then performs asynchronous retrieval in the cooperative system-prompt/assemble waterfall. Its context has the stable name dsh-selective-memory:transparent-window.

• One REMI window is present in an assembly.
• The current DSH runtime-context snapshot supersedes earlier snapshots on the active surface.
• DSH still stores snapshot events in the durable log for replay and auditing.

“Transparent” means that users do not have to copy summaries or manage context blocks manually. It does not mean hidden or unlogged injection.

SQLite schema

Schema v4 contains:

• memories: text, hashed embedding, importance, affect, access counters, source metadata, lifecycle status, validity interval, and supersession links;
• retrieval_traces: complete scoring and selection traces;
• memory_associations: canonical session-local co-activation edges;
• session_scopes: immutable host owner/project bindings;
• content_tombstones: exact-content replay suppression;
• schema_meta: schema version and graph namespace migration flag.

Opening an older database performs additive migrations; v4 adds scopes and tombstones and namespaces legacy session graph keys transactionally. Back up an inactive database before upgrading. See the framework guide for rollback limits.

Telemetry and RetrievalTrace

JSONL event types:

• plugin/started
• memory/ingested
• memory/correction
• memory/retrieval
• memory/window
• memory/emotion-analysis
• memory/compaction
• plugin/error

JSONL telemetry contains a stable query fingerprint, not the raw query or memory text. SQLite retrieval traces include component scores, selection decisions, token counts, query affect metadata, timing, and privacy-preserving graph counters: seeds, edges read/applied/pruned/stored, boosted candidates, reinforcement-eligible memories, and proposed/reinforced pairs.

Benchmark

Run the included dataset:

pnpm benchmark

Run a custom JSONL dataset:

node packages/benchmark/dist/cli.js C:/path/to/dataset.jsonl

Exercise graph priming and the no_hebbian control:

pnpm benchmark:graph

See examples/benchmark/sample.jsonl for the basic input shape and examples/benchmark/graph-ablation.jsonl for a primed graph case. Cases may include primingQueries to build graph state before the measured query. The runner exposes full, no_window_hygiene, no_hebbian, no_affect, similarity_only, no_recency, and no_importance, and reports graph activity separately. Its hitRateAtK is a string-evidence pipeline check, not a complete quality evaluation.

Repository layout

packages/
  core/             # DSH-independent types, scoring, budgeting, compaction, traces
  store-sqlite/     # SQLite MemoryStore and schema migration
  dsh-adapter/      # DSH events, context window, affect calls, compaction, telemetry
  benchmark/        # Replay, paired validation and independent DSH meter
  manager/          # Explicit-ID local management service and CLI
examples/
  deepseek-harness/ # Cordis group and root-overlay examples
  benchmark/        # Minimal benchmark dataset
docs/
  API_COMPATIBILITY.md

packages/core imports no DeepSeek Harness modules and makes no model calls. DSH lifecycle integration and model routing are isolated in packages/dsh-adapter.

Development and verification

pnpm build
pnpm typecheck
pnpm test
pnpm benchmark

Current verification results:

• 13 test files and 81 tests pass on this framework branch (Linux / Node 24);
• every workspace package builds and type-checks;
• an isolated DSH Desktop 2.0.6 headless smoke test returned REMI_LIVE_OK;
• the smoke-test memory window used 470 of 600 estimated tokens;
• custom compaction reduced 1,766 estimated input tokens to 478 output tokens;
• a clean-database DSH Desktop 2.0.6 current-truth run preserved the active value, the superseded value, and an unrelated same-valued fact across native compaction and a Desktop restart.
• in the three-case deterministic benchmark fixture, window hygiene kept hit rate at 1.0 while reducing mean selected memories from 2 to 1 and mean estimated retrieval tokens from 89 to 74 versus no_window_hygiene. This fixture verifies mechanics, not model-answer quality or production cost savings.
• a live DSH Desktop 2.0.6 transparent-window run confirmed wrapped acknowledgements as low-utility, sibling entities as focus-mismatch, explicit future recall with utilityFactor = 1, and instruction-wrapped duplicate facts as one selected plus one redundant candidate.
• a 96-round live long-context run recalled both early facts before and after native DSH compaction; the post-compaction request used 11.6K model-input tokens instead of 172K before compaction, while REMI selected exactly 2 facts from 200 candidates and injected an estimated 132 tokens.

The headless smoke test used a deterministic local provider to verify lifecycle integration and ctx.llm.stream(). The isolated current-truth, transparent-window, and long-context runs used the configured live DSH model path. Provider retries were counted as provider behavior; SQLite lifecycle state and REMI telemetry were checked independently. See the current-truth validation note, window-hygiene validation note, and long-context validation note for the exact evidence boundaries.

Known limitations

• No automatic backfill for sessions created before the plugin was installed.
• No default cross-session retrieval scope.
• Hashed embeddings are placeholders with limited cross-language and synonym recall.
• Explicit sibling-entity isolation is currently strongest for ASCII identifier-like terms; CJK-only entity separation still depends mainly on lexical evidence.
• Current-truth matching is limited to explicit deterministic forms and exact subject/value evidence; there is no model-based resolver or merge UI; explicit-ID deletion is available through the local manager.
• Current-truth has one clean-database live validation run, including native compaction, crash recovery, and an app-controlled Desktop restart. This remains a bounded English protocol, not a broad multilingual or long-duration evaluation.
• Hebbian edges express retrieval co-occurrence, not truth or causality.
• Affect hints have no cross-language calibration, temporal smoothing, or diagnostic meaning.
• Extractive compaction is deterministic but can omit implicit decisions or retain outdated text.
• Token counts are heuristic; actual pressure is determined by DSH ctx.tokenMeter and model capacity.
• DeepSeek Harness is pre-release software. Re-run the upgrade checklist before updating DSH packages.

License

MIT

Early testing notice: Do not use this plugin in production environments.