dsh-langfuse

@amaster.ai/dsh-langfuse

Langfuse observability for DeepSeek Harness (dsh):

• one generation per LLM call (llm/stream waterfall), plus a nested llm-request span carrying the verbatim loop-built request
• one tool observation per tool call (tools/pre-execute through tools/result)
• one span per hook/invoked / hook/result pair, with hook point, decision, exit code, and duration (stderr summary follows captureContent)
• one trace per session turn (session/event) — in the v5 SDK the trace IS its root span, ended (and thereby exported) at turn/end
• subagent child sessions nested under the parent's tree (session/created header link + subagent/start / subagent/end)
• structured reasoning in generation output, optional image display, inherited user/tags, and final turn severity
• optional W3C parent context supplied per session, without adopting the host's ambient OTEL context

Install

dsh plugin --profile my-agent add @amaster.ai/dsh-langfuse \
  @langfuse/tracing @langfuse/otel @opentelemetry/sdk-trace-node \
  @opentelemetry/api @opentelemetry/exporter-trace-otlp-http

The SDK peers (@langfuse/tracing, @langfuse/otel, @opentelemetry/sdk-trace-node, @opentelemetry/api) are loaded via lazy dynamic import(), so an unused install costs nothing — but they must all be present at runtime (dsh plugin add does not auto-install peer trees, so the exporter that @langfuse/otel itself peers on is listed explicitly).

Configuration

Disabled by default. Configure via the profile's cordis.patch.yml:

- insert:
    - id: langfuse
      name: '@amaster.ai/dsh-langfuse'
      config:
        enabled: true
        publicKey: pk-lf-...
        secretKey: sk-lf-...            # secret role
        baseUrl: https://cloud.langfuse.com
        traceName: dsh-turn
        captureContent: true            # set false to record metadata only
        userId: developer
        tags: [coding, experiment-a]
        environment: development
        release: build-42
        redactFields: [password, authorization]
        captureMedia: false              # opt in to uploading images
        maxMediaBytes: 5242880            # total image bytes per display payload
        mediaTimeoutMs: 5000              # attachment read deadline

Explicit profile values take precedence over environment variables. Empty credentials/base URL fall back to LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL (then the EU cloud URL). userId falls back to LANGFUSE_USER_ID. Empty environment/release values use the SDK's LANGFUSE_TRACING_ENVIRONMENT / LANGFUSE_RELEASE defaults. LANGFUSE_TRACING_ENABLED=false (or 0) overrides the profile master switch; environment keys alone do not enable the plugin.

Buffered spans drain on session/flush(session); the exporter shuts down with the plugin fiber. Both checkpoints await pending display projections. Unload also closes outstanding model calls with their collected partial content/usage and marks incomplete calls WARNING, including calls whose stream was never consumed; the original stream can still settle without ending those observations twice. Attachment reads run in the background and never delay the agent's chunk stream or tool return value; a read that exceeds its deadline degrades to image metadata, even if the backend ignores cancellation. The SDK's LANGFUSE_FLUSH_AT, LANGFUSE_FLUSH_INTERVAL, and LANGFUSE_TIMEOUT control exporter batching and network timeouts.

With captureContent: true, both llm-call and llm-request inputs contain every llm/stream request option except the non-serializable AbortSignal. New request fields pass through without a plugin update. For image-bearing requests, generation input is updated to a display projection; llm-request retains the original attachment references. Langfuse's separate generation modelParameters attribute summarizes the known sampling fields; it is not part of either request input. These are dsh-side values, not exact provider HTTP payloads: adapters may add defaults, project messages/tools, or rename fields after this seam. With captureContent: false, the collector keeps chunk/tool-call counts and usage without buffering text, reasoning, tool arguments, or raw chunks; inputs keep structural counts while the separate modelParameters attribute keeps safe parameter metadata; reply excerpts, reasoning, tool arguments/results, error text, and media are withheld from observation names and IO.

Configured Langfuse credentials and pk-lf-* / sk-lf-* values are redacted across reporter payloads, names, status messages, and labels. redactFields replaces matching payload fields recursively, including JSON-encoded arguments, ignoring case. Name excerpts are derived after redaction. Delta blocks are also checked as assembled text so fragmented credentials cannot bypass the mask: affected blocks keep their record count/type/index, with sanitized text in the first delta and later deltas cleared. Redaction only changes telemetry copies; original messages/results and the forwarded stream stay intact. The exporter applies the same mask to IO and metadata as a final safeguard. User/tags are explicit metadata and remain when content capture is off.

Reasoning, images, and turn results

Text-only generation output remains a string. When reasoning or tool calls are present, output uses the Langfuse chat format: role: "assistant", optional content, thinking blocks, and tool_calls with the original call ID and function arguments. This renders thinking and invoked tools in the formatted preview. The original reasoning deltas remain in the nested llm-request output; no extra generation is created.

captureMedia: true requires content capture and an available ctx.attachments service (@deepseek-ai/dsh-attachment >=0.2.0-rc.1). Prompt, generation-input, and tool-output image references are read through that service and converted to data URIs for the SDK's media uploader. Install the attachment bundle appropriate to the deployment; the plugin does not instantiate a storage backend. Missing/failed reads, offloaded images, unsupported MIME types, oversized images, and disabled media fall back to MIME/size/dimension metadata. A generation's byte budget covers all messages in that request. Observation end times are captured when the model, tool, or turn completes; delayed image reads do not extend business latency. LANGFUSE_MEDIA_UPLOAD_ENABLED=false (or 0) overrides media capture. Verbatim file references remain metadata; their bytes are not uploaded.

Root severity follows the final turn reason: errors are ERROR; cancellation, blocking, token limits, and interruptions are WARNING. A recovered intermediate generation/tool/hook/subagent error sets hadErrors metadata while a successful final turn remains DEFAULT. This keeps retry failures visible without treating recovery as a failed turn. Outstanding tools also close as WARNING at unload, including calls without an agent/session; late results still reach their original callers without repeated telemetry or media reads.

Per-request metadata, labels and distributed tracing

Hosts can supply context before a session's next trace starts:

ctx.emit('langfuse/context', session.id, {
  userId: 'user-123',
  tags: ['support', 'experiment-b'],
  metadata: {
    projectId: 'project-a',
    requestId: 'request-123',
    experiment: { name: 'retrieval', variant: 2 },
    labels: ['interactive'],
  },
  traceparent: '00-1234567890abcdef1234567890abcdef-1234567890abcdef-01',
});

The typed event is exported through this package's Cordis augmentation. Context is consumed once by the next turn trace (or an out-of-turn trace created first); send it again for another request. Labels override profile defaults for that trace and propagate through generations, tools, hooks, and nested subagents. Queued context is discarded when the session is disposed or the plugin unloads. Concurrent sessions have independent context; no process-global request environment is modified.

metadata accepts a plain JSON object with arbitrary string keys and nested values: plain objects, dense arrays, finite numbers, booleans, null and strings (including empty strings). The event validates, snapshots and masks the context immediately, so later caller mutations cannot change it. Request metadata is carried by every observation; plugin-generated fields such as turn, provider, toolName and taskId take precedence on that observation. Existing credential masking and redactFields apply, and explicit metadata remains available when captureContent: false.

Non-JSON metadata rejects the entire submitted context with a diagnostic, without interrupting the agent or retaining an earlier pending context. This includes non-finite numbers, undefined, functions, symbols, BigInt, circular references, throwing getters, class instances (including Date, Map and Set), sparse arrays, extra array properties, and symbol-keyed or non-enumerable object properties. Values are never silently converted or dropped, and custom toJSON functions are not invoked. Omitted optional context fields remain supported. For an existing A2A task, supplying new metadata replaces its request metadata for the task root and subsequently created turns and their descendants; already-created observations retain their context.

The plugin preserves JSON types in its OTLP payload. Langfuse server APIs may normalize the stored values: for example, v3's read API parses a top-level metadata string "123" as the number 123.

Only explicit W3C version-00 parents with non-zero trace/span IDs are joined. Invalid headers log a prefixed diagnostic and start an independent trace. Without a header the plugin always starts an independent trace. Each observation is created inside native OTEL ROOT_CONTEXT: foreign spans and Langfuse baggage (user/tags/metadata) are excluded, while explicit parents and plugin labels are preserved. This scoped context does not replace the host's global provider or context manager. When joining an external trace, dsh's turn is an observation under the supplied parent; the SDK's automatic app-root claim is disabled and the external owner's trace name and trace-level IO stay intact.

Token usage and cost

Generation usageDetails has disjoint input, cache_read_input_tokens, cache_creation_input_tokens, output, and output_reasoning_tokens buckets plus total. Reasoning is subtracted from provider output before becoming its own bucket; cached input is counted separately. These keys match the official Pi integration. Custom definitions using the previous input_cache_read, input_cache_creation, or output_reasoning names must update their price keys. This avoids double-counting both usage and inferred cost. On Langfuse v3, the compact token badge may display only the base input/output buckets. Open its usage breakdown for aggregate input (including cache) and output (including reasoning); total already includes every bucket. Do not add cached tokens to input again to change that badge.

Langfuse infers USD cost when the generation model matches a model definition with prices for the exact usage keys. In Project Settings → Models, configure these buckets for private routes, model aliases, or custom prices; reasoning output generally uses the model's output rate. A token total does not prove that a matching price definition exists. If a generation shows “Create model definition”, its model has no matching definition. For gateway aliases, use the actual routed model and your gateway’s rates when defining prices; the plugin keeps the requested model name rather than guessing the upstream model. Definition changes apply to new generations only. dsh's current TokenUsage does not carry billed cost, so the plugin does not claim provider-invoiced costs or maintain its own price table.

Compaction and session-title calls passing through llm/stream already receive their own generation and purpose metadata, including out-of-turn calls. Tools using the same LLM service are observed there too; adding a second generation from tool-result usage would duplicate accounting.

Hook spans appear when a dsh hook bridge emits the paired session records. With content capture enabled, they show hook::, optional input and parsed result, and available bounded process output alongside execution status. Producers must emit these fields; upgrading this plugin cannot recover omitted data. SessionStart only appears if a producer reports it and the installed runtime admits the records.

Custom hook producers

Custom bridges use the same hook/invoked and hook/result session records. The writer and consumer accept an unrestricted dialect source string (gemini, codex, company-validator, etc.); admission to the session log still depends on installed runtime invariants. No dialect registration or separate exporter is needed. The plugin observes records only and never executes hooks or changes their decisions.

Import appendHookEvent, HookInvoked, and HookResult from @amaster.ai/dsh-langfuse. This typed writer uses the real session.append publication path without redeclaring SessionEventMap. Its TypeScript adaptation allows custom payloads despite the optional upstream dsh-hook-protocol type augmentation; it does not widen that package's runtime contract, runner, or bridge APIs. The writer validates fields, catches append failures, logs [dsh-langfuse], and returns a boolean; telemetry success must never control a hook's decision.

DSH 0.2.1-rc.2. The upstream hook invariant companion and dsh-invariants package are no longer part of this runtime. Custom dialects and delayed results are accepted through the session publication seam; the plugin still validates hook records and contains append failures.

Runtime compatibility boundary (verified with `dsh-hook-protocol@0.2.0-rc.2`). When its invariant companion is registered and enabled through dsh-invariants, it rejects dialects other than claude-code/codex, invocations outside an open turn, and results after their turn closes or while another turn is open. A type assertion cannot bypass these pre-commit checks. appendHookEvent returns false for rejected records; they are not committed or published and produce no Langfuse hook observation.

Runtime composition · Claude Code/Codex pairs in the open turn · Custom dialects · Detached invocations / late results No active hook-protocol invariant · Supported · Supported · Supported with invocationId Active hook-protocol invariant 0.2.0-rc.2 · Supported · Rejected · Rejected

The writer never disables or reconfigures invariants. Applications using the active companion need an upstream contract change before adopting custom or late records. The producer still owns hook execution and decisions even when telemetry is rejected. The following custom example requires a runtime composition that admits those records.

import { randomUUID } from 'node:crypto';
import { appendHookEvent } from '@amaster.ai/dsh-langfuse';

// The bridge determines turn/point, executes the command, and owns its outcome.
const invocationId = randomUUID();
const identity = { turn, point: 'BeforeAgent', handlerId: 'request-validator', invocationId };
appendHookEvent(session, 'hook/invoked', {
  ...identity,
  dialect: 'company-validator',
  name: 'Validate request',
  input: { request: 'example' },
});
// After execution, including exceptions, timeout or cancellation:
appendHookEvent(session, 'hook/result', {
  ...identity,
  decision: 'pass',
  outcome: 'success',
  exitCode: 0,
  durationMs: 37,
  result: { reason: 'valid', additionalContext: 'checked', continue: true },
});

Every custom execution should have a unique invocationId and a result emitted in the producer's finally path. Both records must share turn, point, handlerId, and invocationId. handlerId identifies the configured handler; invocationId identifies one execution, including concurrent executions of that handler. Optional name labels the observation when content capture is enabled. HookInvoked.input carries the script payload; HookResult.result carries the parsed output object, including available reasons, additional context and continue/stop instructions. The consumer preserves these producer-defined fields without interpreting them as decisions. Unmatched/skipped hooks must not emit an invocation: these records represent actual execution. Optional stdout, stderr and legacy stderrSummary are masked, then limited to 4096 characters each in observations, with stdoutTruncated, stderrTruncated and stderrSummaryTruncated metadata. Producers should also bound persisted process output and omit unavailable optional fields rather than writing undefined into the JSON session log.

Tool hooks should include the execution's callId in both records. The plugin matches it within the originating session and turn; it never chooses the last active tool. Tool observations begin at tools/pre-execute (a prepended observer) and end at tools/result, covering synchronous/awaited BeforeTool and AfterTool hooks, pre-dispatch denials, and final post-policy output. Bridges must run BeforeTool within the pre-execute waterfall and AfterTool within post-execute, before the final result notification. A bridge registered with prepend ahead of this observer must delegate before reporting its hook. Agent hooks and legacy records without callId remain under the turn/subagent. Unknown or already finished calls fall back to that parent; late invocations outside the originating turn use the detached-root behavior below.

hook/result.outcome describes process execution independently of decision: success is normal completion, failed is execution failure, timed-out is deadline termination, and cancelled is cancellation. The producer maps its process semantics to these values; an exit code expressing a policy denial can still be a successful execution. Optional exitCode and signal preserve process details. Failed/timed-out executions receive ERROR observations, cancellation receives WARNING, and successful executions retain decision-based warnings. Records without an explicit outcome preserve the legacy exit-code interpretation (0/2 are non-error). The plugin records durationMs and never infers a timeout or cancellation from a missing exit code.

For records admitted by the session, runs with invocationId retain correlation after turn/end and a subsequent turn/start; a late result ends the original hook span without attaching it to the new turn. Delayed invocations, including UserCancel, reuse their originating turn's parent span even after the next turn starts. The plugin retains the latest 128 completed-turn associations across all sessions, clearing a session's associations on disposal and all associations on unload. Reporting never reopens the ended root or changes its end time or final status; each hook records its own duration. When the originating parent is unavailable or evicted, the invocation receives a separate root with detachedHook: true and the original turn number; its result ends both hook and fallback root. Neither path consumes langfuse/context queued for the next turn. Detached producers must finish reporting before the session is disposed. Session disposal/plugin unload closes unfinished runs with hook result not observed; this means missing telemetry, not proof of execution failure. Legacy records without invocationId still close at turn boundaries.

Custom names, hook input, parsed results and raw process output follow captureContent; credentials and redactFields are masked by the existing reporter. New outcomes, correlation IDs, dialect, point, duration and process metadata remain available with content capture disabled. The event writer does not sanitize the session log itself: application producers remain responsible for what they persist. Applications must explicitly replace any private hook events with this standard producer path, reuse the same invocationId, and report the actual outcome; installing/upgrading the plugin does not adapt an application bridge or repair historical Trace overview IO (Issue #59).

Observability is a no-throw seam: backend errors are logged with a [dsh-langfuse] prefix and never escape into the agent loop.

Compatibility

Run pnpm test:sdk at the repository root for a secrets-free check using the real Langfuse SDK and the existing local OTLP receiver. It verifies export, native tool type, SDK label redaction, ambient-baggage isolation, external trace ownership, agent-less tool unload, the real tool runtime and hook processes, tool-hook parenting and enclosing timestamps, content masking, and custom hook export across turn closure. It is separate from the mocked unit suite and the gateway/real-Langfuse E2E legs.

The independent dsh-langfuse.tool-hooks E2E leg boots the packed plugin in the real headless profile, asks the model to read a marker file, and executes BeforeTool/AfterTool scripts through an E2E-only producer. It requires both gateway and Langfuse credentials, then polls the real Observations API to verify named hooks, tool-call parenting, enclosing tool timestamps, script input and parsed output, credential masking, and stdout truncation. The existing dsh-langfuse.hook leg continues to check the upstream Codex UserPromptSubmit bridge; the new fixture verifies the shared event contract and does not substitute for the application's Gemini bridge integration.

The dsh-langfuse.cancel-hook E2E leg cancels a real agent at a tool boundary, queues its next turn, and asynchronously runs a UserCancel process after that turn starts. Real Langfuse API read-back must show the hook under the cancelled turn's root and trace ID, with the root still aborted and ended before the hook's own duration. It requires the existing gateway and Langfuse secrets in integration.yml.

Trace overview compatibility

Overall input/output belongs to the dsh-turn root observation. The root is exported when the turn ends; while a long turn is running, child observations can already be visible before the root appears. After completion, allow asynchronous ingestion to settle.

The plugin also sends deprecated langfuse.trace.input / langfuse.trace.output attributes on independently owned roots for legacy Trace overview compatibility. It never writes these fields when joining an externally owned trace. Langfuse recommends root-observation IO; empty Trace overview fields alone do not establish missing observation data.

Real Trace/root API read-back was verified on self-hosted Langfuse 3.172.1 with JS SDK 5.10.1 and 5.11.1. This verifies new synthetic turns, including children exported before the root and an external observation joining the trace. It does not establish compatibility for every server version or repair historical traces. Issue #59 contains a historical SDK 5.11.1 trace whose root IO is present but Trace IO remains empty; the cause of that discrepancy has not been reproduced on new turns. For that trace, select the dsh-turn observation to read the overall request and response.

To verify your deployment without an LLM, set LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL in the process environment, then run:

pnpm --filter @amaster.ai/dsh-langfuse build
node .github/scripts/integration/