dsh-hindsight-daemon

dsh-hindsight-daemon

English | 简体中文

A Hindsight memory daemon lifecycle plugin for DSH Desktop.

It starts the Hindsight daemon when DSH starts and stops it — together with the embedded PostgreSQL it depends on — when DSH exits. Nothing stays resident: no always-on process, no autostart entry, no scheduled task, no background watcher.

Once installed, Settings → Hindsight memory gets its own page where you can inspect status, install dependencies, start and stop the service, and change the configuration.

Supports Windows 10/11 and Linux (x86_64). Windows installs through PowerShell scripts, Linux through scripts/install.sh; the teardown lands on taskkill and pg_ctl / pkill respectively.

Features

• Lifecycle is the contract. DSH running = daemon running. DSH closed = every process gone. No scheduled task, no resident watcher.
• Three exit paths. A Cordis disposer, a process.on('exit') hook, and signal hooks — any one of them is enough to shut down cleanly, which is what it takes to cover both the Windows desktop host and the Linux CLI.
• A dedicated settings page. Service status and control, environment detection, one-click dependency install, current memory usage, multi-repository (profile) create/switch/delete, LLM endpoint configuration with a connectivity test.
• Cross-platform. One code path for Windows and Linux, branching only in lib/core/platform.js.
• Zero runtime dependencies. Config I/O, the settings UI and the HTTP routes are all self-contained, so nothing breaks when DSH internals move.
• Self-protecting. If the plugin fails to initialise it degrades (no settings page) instead of preventing DSH from starting.

Why this plugin exists

Hindsight's coding-agent integration has three modes: cloud, self-hosted and daemon. In daemon mode memory lives on your machine (embedded PostgreSQL plus pg0) and only fact extraction calls out to an LLM — your data never leaves the box and no cloud account is needed.

But the upstream plugin has two limitations in daemon mode that stop it from starting the daemon itself:

1. It only reads the process environment. detectLlm() inspects the DSH process's process.env (HINDSIGHT_API_LLM_PROVIDER, OPENAI_API_KEY, ANTHROPIC_API_KEY, …), so it never sees the values already sitting in ~/.hindsight/profiles/<profile>.env. Detection fails and the log records daemon mode needs an LLM for fact extraction.
2. It only waits 12 seconds. At session start seedIfCold waits DAEMON_WAIT_SESSION_START_MS = 12e3, and it tries once per process. A cold daemon start takes about 30 seconds (import torch plus import sentence_transformers alone accounts for 12–13 s), so it gives up long before the daemon is healthy.

This plugin takes daemon startup away from the upstream plugin and does it inside DSH's own Cordis tree: earlier in the boot sequence, without an extra process, and with a clean teardown on exit.

How it works

DSH starts
  └─ Cordis loads this plugin (lib/index.js → apply)
       ├─ registers /hindsight-daemon/api/* routes (for the settings page)
       └─ spawns in the background: hindsight-embed -p <profile> daemon start
            └─ daemon (pythonw) listens on 127.0.0.1:9077
                 └─ pg0 starts embedded PostgreSQL (:5432)

DSH exits (three paths, any one of them suffices)
  ├─ (1) Cordis disposer: the desktop main.js disposes the tree in before-quit (5 s grace)
  ├─ (2) process 'exit' hook: normal termination, or signal termination when a runtime
  │       such as signal-exit is present
  └─ (3) SIGTERM / SIGHUP / SIGINT hooks: adopted only when the host registered none
       └─ teardown (fully synchronous — a hard constraint of signal hooks):
            ├─ kill the detached hindsight-embed CLI
            ├─ find the daemon PID by port and kill it
            └─ pg_ctl stop -m fast -D <data>   ← graceful shutdown, no WAL replay next boot

The three paths are entrances only; the work is one idempotent cleanup (shared.closed lets it run once).

Measured: apply() returns in about 2 ms (DSH startup is not blocked), cleanup takes about 500–600 ms, comfortably inside the 5 s grace. Cold start to healthy is about 20 s on Linux with models and dependencies in place.

Why three paths: different hosts give different exit guarantees.

• The Windows desktop host intercepts quit, then disposes the tree — path 1 is enough.
• The Linux dsh CLI registers no signal handlers at all (measured: process.listenerCount('SIGTERM') is 0). kill terminates it outright and path 1 never runs.
• Real processes usually already have signal listeners from another library (signal-exit, for example), which fire Node's exit event before termination — so path 2 is the one that actually works on Linux. Path 3 only takes over when nobody handles the signal, so it never races the host's asynchronous teardown.

Cleanup has to do all three things explicitly, each for a reason learned the hard way:

• hindsight-embed daemon start is detached and respawns the daemon after it is killed — so the CLI goes first.
• hindsight-embed daemon stop does not stop PostgreSQL and takes about 4 s, uncomfortably close to the 5 s grace — so the daemon is killed by port-derived PID and the database is closed separately with pg_ctl stop -m fast (about 160 ms).

The full process topology, the reasoning behind each path, and debugging notes live in docs/architecture.md.

Requirements

Install

Option 1: one-shot script (recommended)

git clone https://github.com/FYTXY/dsh-hindsight-daemon.git
cd dsh-hindsight-daemon

Windows (PowerShell):

powershell -ExecutionPolicy Bypass -File .\scripts\install.ps1

install.ps1 appends the insert block and then calls scripts/install-deps.mjs to install the Python dependencies and the two local models. Pass -SkipDeps to skip that and use the settings page instead.

Available parameters:

-Profile <name>   DSH profile name, defaults to desktop
-DshHome <path>   DSH home directory, defaults to $env:DSH_HOME or %USERPROFILE%\.dsh
-SkipDeps         Only mount the plugin; install dependencies from the settings page

Linux:

chmod +x ./scripts/install.sh
./scripts/install.sh                     # installs into the web profile
./scripts/install.sh --profile desktop   # or name a profile

install.sh does one thing: append the insert block to the profile's cordis.patch.yml, using the absolute path of this repository's lib/index.js so a git pull takes effect immediately. Dependencies are left to the settings page.

Both scripts are idempotent: running them twice does not add a second insert block.

After installing, restart DSH and open Settings → Hindsight memory to enter your LLM endpoint.

Option 2: manual mounting

Put the plugin directory anywhere, then append to <DSH home>/profiles/<profile>/cordis.patch.yml:

- insert:
    - id: hindsight-daemon
      name: '/absolute/path/to/dsh-hindsight-daemon/lib/index.js'

Use forward slashes in name (on Windows too: C:/path/to/...), pointing at lib/index.js. Then install dependencies from the settings page.

Option 3: npm

dsh plugin --profile desktop add dsh-hindsight-daemon

package.json declares dsh.bundle.patch, so the bundle's own cordis.patch.yml mounts the plugin.

Pick exactly one of the three. Mounting twice registers the /hindsight-daemon/api prefix route twice and DSH fails to boot with webserver: duplicate prefix route.

Configuration

Settings → Hindsight memory (placed towards the end of the navigation) has six blocks.

Every button reports back. Clicking one immediately turns its label into a progress state (Saving…, Starting…, Switching…) and disables it, so it cannot be double-fired. Results appear directly under the Save / Re-detect pair: success is a green receipt (for example “Configuration saved”) that clears after five seconds, failure is red and stays until the next successful action.

Service status

Shows phase (stopped, starting, running, installing, error), address and uptime, with Start and Stop buttons. Refreshes every 2 seconds.

Environment detection

Reports readiness for uv, hindsight-embed, PostgreSQL and the database directory. Anything missing is covered by the one-click install. Detection order: configured path > environment variable > PATH > well-known install locations.

One-click install

Idempotent, safe to repeat. Five stages:

The --with hindsight-api part matters: it puts the API in the same venv so the daemon finds hindsight-api.exe directly. Otherwise it falls back to uvx hindsight-api@<version> and re-downloads 600 MB into a temporary environment.

Current memory

Read-only. Answers “where does memory actually live”: database directory, disk usage, and the node / edge / document counts of every memory bank.

One code repository maps to one memory bank, named after the repository, and projects cannot see each other's memories. These values are not editable here — the location is decided by Advanced below, and moving it is described under Where files live.

Memory repositories (profile)

Lists every profile on this machine, marks the active one, and allows creating, switching and deleting.

A profile is an independent repository: its own database, its own env file (including the gateway key), its own log. Creating one only writes an env file — no database, no process — the database appears when the profile is activated and the daemon first starts, so creating one costs no disk. Activating stops the current daemon and starts the new one (about 30 s); the previous repository is left untouched and can be switched back at any time.

Names may contain letters, digits, dots, underscores and hyphens, must start with a letter or digit, and are at most 32 characters. Switching is only allowed to an existing profile — typing a misspelled name would silently create an empty database, which is harder to notice than an error.

Delete sits to the right of each repository (after Activate) and asks for confirmation first. Deletion is irreversible: it removes the database directory, the env file (with the gateway key), the logs and the registry entry. The button is disabled for the repository currently in use — switch to another one first. Deleting never touches other repositories and never stops the daemon.

LLM

Once Base URL, model and key are filled in, Test connectivity sends one minimal request with the current draft and reports reachability, the failure reason, the latency and the model name the endpoint actually echoes back — so a misconfiguration surfaces now rather than during the first memory write. An empty key falls back to the saved one, so testing right after typing works. The test is read-only: it writes no configuration and does not disturb a running daemon.

Advanced

The first two decide where memory lives; the last three only influence downloads and listening — which is why the dangerous ones come first and carry their own caveat.

What is a memory bank called

The upstream plugin derives the bank name from a template, coding-agent::{gitProject} by default, where {gitProject} is the name of the git repository root:

Available placeholders: {gitProject} (git root name, worktrees resolve to the main repository), {project} (current directory name, no repository resolution), {harness}, {channel}, {user}.

So switching to another directory that is another git repository automatically writes into another bank — no configuration needed, no cross-contamination. An empty bank just means you have not talked in that repository yet.

The bottom of the page merges the plugin log and the daemon log (tails of each); start there when debugging.

Where files live

~ is the home directory: %USERPROFILE% (C:\Users\<you>) on Windows, $HOME on Linux.

Uninstall

powershell -ExecutionPolicy Bypass -File .\scripts\uninstall.ps1

chmod +x ./scripts/uninstall.sh
./scripts/uninstall.sh                     # removes from the web profile
./scripts/uninstall.sh --profile desktop   # or name a profile

Removes only the insert block from cordis.patch.yml (backed up first). After a restart the plugin no longer loads. Python dependencies, models and the database are kept so you can mount the plugin again later; to clean those up completely, delete the directories listed above.

Troubleshooting

First start is slow. Expected. The first run downloads 1.5 GB of dependencies plus 1.2 GB of models; afterwards a cold start is about 30 seconds, mostly Python import time.

daemon mode needs an LLM for fact extraction in the log. That is the upstream @vectorize-io/hindsight-coding-agents plugin complaining that it cannot detect an LLM — it only reads the DSH process environment. This plugin does not use that path; ignore the line as long as the daemon status is running.

Dependency installation fails halfway. It is network-sensitive. The Tsinghua PyPI mirror and hf-mirror are used by default; behind a corporate proxy point the mirrors at internal addresses or blank them out for the official sources. Retrying after a partial failure is safe — downloaded wheels are cached.

Port 9077 is taken. Change Local port and restart DSH. Before starting, the plugin probes /health; if a healthy daemon already listens there (perhaps one you started manually) it is reused rather than duplicated.

Port or profile name changes have no effect. Both require a DSH restart (they rebuild the profile and the global configuration).

The daemon survives an unclean DSH exit (crash, force kill). No teardown path ran (SIGKILL cannot be intercepted by any language). Collect it manually:

# Windows
taskkill /F /IM hindsight-embed.exe /T
Get-NetTCPConnection -LocalPort 9077 -State Listen | ForEach-Object { taskkill /F /PID $_.OwningProcess /T }
& "$env:USERPROFILE\.pg0\installation\*\bin\pg_ctl.exe" stop -m fast -D "$env:USERPROFILE\.pg0\instances\hindsight-embed-<profile>\data"

# Linux
pkill -f hindsight-embed
pgrep -f "hindsight" | xargs -r kill -9          # or by port: ss -ltnp | grep 9077
"$HOME/.pg0/installation/"*/bin/pg_ctl stop -m fast -D "$HOME/.pg0/instances/hindsight-embed-<profile>/data"

If the database was not shut down cleanly, PostgreSQL replays the WAL on the next start. No data is lost.

Which exit path actually fired? Start DSH with DSH_HINDSIGHT_DEBUG=1 and the plugin writes hook registration and cleanup decisions to ~/.dsh/hindsight-daemon/debug.log:

exit hooks registered pid=58065 SIGTERM=2 SIGINT=2 exit=3
received SIGTERM, listeners=2 pid=58065
cleanup(exit) enabled=true stopOnExit=true busy=null

The listeners count matters: above 1 means the host or a dependency already owns the signal, the signal hook steps aside, and cleanup runs through the exit path instead.

An upgrade did not change behaviour. Most likely a second copy is loaded. A DSH profile has two mounting routes: dsh.profile.bundles (resolved by package name from node_modules) and the profile's own cordis.patch.yml (which may hold an absolute path). An absolute path left in cordis.patch.yml overrides the bundle entry, so the version installed by dsh plugin add is never loaded. Check:

grep -rn "hindsight" ~/.dsh/profiles/<profile>/cordis.patch.yml

Keep only one route, ideally with name set to the package name (dsh-hindsight-daemon) so the version follows dsh plugin add.

Development

Zero runtime dependencies, nothing to npm install:

npm test          # node --test, the whole suite
npm run check     # node --check on every entry point
npm run pack:dry  # inspect the published file whitelist

No test needs DSH or a real daemon: they redirect HOME, USERPROFILE and DSH_HOME into a temporary directory and run the browser half against a minimal hooks implementation. Conventions are in CONTRIBUTING.md.

Running one file:

node --test tests/units.test.mjs
node --test tests/packaging.test.mjs

Debugging the installer outside DSH:

node scripts/install-deps.mjs

Repository layout

lib/
  index.js        Cordis entry: when to mount routes, start the daemon, tear down
  lifecycle.js    Lifecycle: detect -> start -> stop -> install -> save config
  api.js          /hindsight-daemon/api/* routes plus the Host allowlist
  config.js       Config I/O (atomic write, secret redaction, flattening)
  client.js       Browser half: the settings page
  core/
    paths.js      Disk probing: uv / hindsight-embed / pg_ctl / data directory
    daemon.js     Health probing, PID by port, spawn, three-step shutdown
    profile.js    Profile creation, line-level env patching, global config
    profiles.js   Profile listing / create / delete (with directory size cache)
    installer.js  Five-stage install (streamed child output)
    platform.js   Platform branch: executable names, venv paths, process control
    llmTest.js    Minimal request used by the connectivity test
    memoryInfo.js Database size and memory bank statistics
    status.js     Status object and the log ring buffer
scripts/
  install.ps1 / install.sh     One-shot install (writes cordis.patch.yml)
  uninstall.ps1 / uninstall.sh Remove the insert block
  install-deps.mjs             Dependency and model install outside DSH
tests/                         node:test suites (no DSH required)
docs/architecture.md           Process topology, exit paths, trade-offs

The zero runtime dependencies decision is deliberate: configuration is read and written directly, the settings page is rendered directly, and nothing imports schemastery, dsh-shared or dsh-api-gateway. Those packages are published on npm at a different version line (0.0.1-rc.1) from the ones DSH bundles (0.2.0-rc.2), so depending on them would break the plugin whenever DSH updates.

Compatibility

• DSH Desktop 2.0.x, the web profile and the desktop profile alike.
• Cordis 4 (apply() returning a disposer, ctx.get('webServer') for route registration).
• Node.js 22 and 24 are exercised in CI on Windows and Linux.

Security

Everything stays on loopback: the daemon listens on 127.0.0.1:9077, and the plugin's own routes sit on DSH's web server behind a Host-header allowlist (127.0.0.1, localhost, [::1], plus the host's own trusted hosts). Nothing binds to an external interface.

The LLM API key is stored in plain text in <DSH home>/hindsight-daemon/config.json and in ~/.hindsight/profiles/<profile>.env; filesystem permissions are the boundary. The settings page never echoes it back (it shows •••• and an apiKeySet flag).

Found a vulnerability? Please report it privately rather than in a public issue — see SECURITY.md.

Contributing

Development setup, test conventions and compatibility rules are in CONTRIBUTING.md.

License

MIT