PerryLink/dsh-data-quality

dsh-data-quality

Deterministic data profiling, cleaning, and verification for DeepSeek Harness.

• 1024 store channel: npm i -g dsh1024 once, then dsh1024 plugin --profile web add dsh-data-quality (counts toward the deepseek1024.com install ranking).

Deterministic data profiling, cleaning, and verification for DeepSeek Harness.

All computation is plain TypeScript in the harness process — the model never does the math. A ctx.dataQuality capability seam (Service Definition / local Provider / tool Consumers) exposes three model tools plus a frozen cross-plugin citation-checking contract.

English · 简体中文 · Español · Português · हिन्दी

dsh plugin --profile <name> add dsh-data-quality

📖 Ecosystem knowledge base — measured data, not marketing: plugin development guide · plugin-selection data · maintenance criteria.

Install & uninstall

dsh plugin --profile web add dsh-data-quality      # install (npm) — or the forms above
dsh plugin --profile web remove dsh-data-quality   # uninstall

⭐ 如果它帮到了你

这个插件是 DSH 插件家族的一员（95 个，全部 Apache-2.0）。如果你在用，给个 star —— 它不会解锁任何功能，但会让下一个人在搜索里更容易找到它。

*English:* part of a 95 plugin family for DeepSeek Harness. If it is useful, a star helps the next person find it — nothing is gated behind it.

What is dsh-data-quality?

Deterministic data profiling, cleaning, and verification for DeepSeek Harness.

*Workflow overview: question → sealed sources → report. The examples below use the real command surface.*

Comparison

fixtures/dirty-fund-holdings.csv (10 rows x 6 columns) · 5 determinable dimensions · run locally

Compatibility

Component · Version DeepSeek Harness · dsh-v0.2.1-alpha.1 (adapted 2026-10-04): the peer range now admits the alpha.2 line; there Session.append's third parameter exists only for surface-eligible types and is a SurfaceIntent, so the audit gate still skips and the storage-domain report stays the durable copy. Verified 2026-09-24 (dual typecheck rulers + full test suite green). Node.js · ^22.19.0 \ · \ · >=24.0.0 Package manager · pnpm@11.7.0 Platform · Windows / macOS / Linux (host-only plugin)

What you get

• `ctx.dataQuality` service — a Cordis service other plugins may optionally consume (inject = ['dataQuality']). Besides the three dataset operations behind the tools, it implements the frozen verifyCitations(request) contract: verify that numbers/strings cited in a document match a dataset snapshot, with relative-tolerance numeric comparison and verified / mismatch / not-found / unverifiable statuses.
• `data_profile` tool — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table sha256 content-hash duplicate detection with the duplicate rate and a bounded sample of duplicate row indexes. Adds a deterministic DAMA six-dimension scorecard (completeness, uniqueness, validity, consistency, timeliness, accuracy — accuracy is reported undetermined without a declared schema, never fabricated). Optional deterministic systematic sampling for large files.
• `data_clean` tool — ordered declarative cleaning rules: dedupe (by column group), fill-missing (constant/mean/median/forward), coerce-type (number/date/boolean; failures counted and set to missing), normalize-unit (e.g. 万/亿 suffixes to base units), trim, map-values (enum mapping). Returns a per-rule audit log, a pre-delivery contract validation summary (dedupe before/after, uniqueness, non-null and type regressions), and a bounded preview; writes the cleaned dataset only when outputPath is given, and never overwrites the source.
• `data_verify` tool — declarative verification rules: not-null, unique, range, regex, enum, cross-column (e.g. startDate < endDate), freshness (date column within N days of a reference date). Per-rule pass/fail with capped failing-row evidence; an overall failure is a normal passed: false result, not a tool error.
• `data_report` tool — read persisted reports back from the data_quality storage domain: by exact reportKey (path-safe validation, missing keys fail loud) or by kind (chronological listing). Returns the report envelope(s) — kind, dataset, timestamp, and the full stored report. format: html renders one profile/clean report as a self-contained offline HTML document (inline CSS/JS, no external requests) with the DAMA six-dimension scorecard and the profile/cleaning summary tables.
• Durable reports — every profile/clean/verify/citation run persists to the data_quality storage domain (JSON backend), keyed by run timestamp plus a dataset-path fingerprint; the key is returned as reportKey in tool results. Clean reports also persist the bounded preview and the contract summary, so every model-visible result is reconstructable from its reportKey; each clean run additionally persists a clean-diff before/after profile report.
• Session events — on hosts that can carry them safely, runs append data-quality/profile / data-quality/clean / data-quality/verify events (with the ignorable marker where supported). On the published 0.1.7-rc.2 line (as on earlier rc lines) the append is skipped by design — the storage-domain report is always the durable copy (see "Known limitations").

Quick start

dsh plugin --profile web add github:PerryLink/dsh-data-quality

npm channel

dsh plugin --profile web add dsh-data-quality

Tarball channel (no build permission needed)

pnpm pack                                  # produces dsh-data-quality-<version>.tgz
dsh plugin --profile web add ./dsh-data-quality-<version>.tgz

Git channel

dsh plugin --profile web add github:YOUR_ORG/dsh-data-quality#<commit-sha>

The first add fails because pnpm blocks the package's prepare build; copy the exact key pnpm printed into the profile's pnpm-workspace.yaml and re-run:

allowBuilds:
  'dsh-data-quality': true

Restart the profile after installing (bundles activate on restart). Then ask the agent, in a workspace containing a CSV:

Profile holdings.csv, then clean it by trimming whitespace, deduplicating on fund_code, and normalizing the holding_value column's 万/亿 units; finally verify fund_code is unique and not null.

Examples

These are the plugin's real entry points; install once, then call one.

Detailed example

Command

~~~sh data_profile({ path: "dirty-fund-holdings.csv" }) ~~~

Output

~~~text Profile of dirty-fund-holdings.csv: 10 rows x 6 columns Duplicate rows: 1 (10.0%); sample row indexes: 8

• nav: mixed; missing 1 (10.0%); unique 7; note: mixed types among 9 present values: 8 numeric, 1 string

Scorecard (overall 93.2%, weighted 93.2%): ~~~

*Observed from README.md:147-149; test/profile.spec.ts:130-132; test/loader.spec.ts:98-99; src/profile.ts:311-315*

Failure branch

Input

~~~text {"path":"dirty-fund-holdings.csv","rules":[{"rule":"dedupe","columns":["nope"]}]} ~~~

Output

~~~text isError=true; error.message=rule 0: unknown column "nope" (columns: fund_code, fund_name, nav, nav_date, holding_value, currency) ~~~

*Observed from plugins/dsh-data-quality/src/clean.ts:66-68; plugins/dsh-data-quality/fixtures/dirty-fund-holdings.csv; exercised through plugins/dsh-data-quality/test/harness.ts*

Brief examples

• data_clean
• data_verify

Configuration

All keys are optional (defaults shown); invalid values fail loudly at load. Every key is settable from cordis.yml (the bundle ships cordis.patch.yml with the same defaults).

Key · Default · Description enabled · true · Master switch; false mounts nothing at all. maxRows · 200000 · Hard row cap per dataset load; larger inputs reject loudly (use the tool's sample parameter). maxFileSizeMB · 64 · Hard file-size cap in MiB per dataset load. defaultTolerance · 1e-9 · Default relative tolerance for numeric citation comparison when a citation omits tolerance. evidenceRowLimit · 20 · Cap on failing-row evidence (verify) and preview rows (clean) in one result. allowedExtensions · ['.csv', '.tsv', '.json', '.jsonl'] · Extensions accepted as datasets. workspaceRoot · "" · Absolute root for SERVICE-level calls (e.g. verifyCitations) that carry no session workspace; empty = the harness process launch directory. Tool calls always use the session's workspace cwd. storeReports · true · Persist run reports to the data_quality storage domain and return reportKey. scorecardWeights · all 1 (equal) · Per-dimension weights (completeness/uniqueness/validity/consistency/timeliness/accuracy) for the scorecard's weighted overall total; each weight must be a non-negative number.

Tools & surfaces

data_profile({ path, sample?, industryPreset? })

Profiles a workspace dataset. path is workspace-relative (.csv/.tsv/.json/.jsonl; JSON must be an array of flat objects). sample takes every ceil(N/sample)-th row for the column cards (deterministic; row counts stay exact). industryPreset (retail/saas/fund/real-estate/e-commerce/healthcare/logistics/manufacturing/energy) injects that industry's expected columns so the scorecard accuracy dimension becomes determinable; unknown ids fail loud. Returns the structured report — the duplicate rate, bounded duplicate-row indexes, numeric count/distinct distributions, file encoding (UTF-8 BOM + validity), and the weighted six-dimension DAMA scorecard — and renders a human-readable per-column summary plus scorecard.

data_clean({ path, rules, outputPath?, dryRun? })

Applies rules in array order, each seeing the previous rule's output. Rule reference:

Rule · Extra fields · Semantics dedupe · columns? · Remove rows whose key-column values duplicate an earlier row (first kept; all columns when omitted). fill-missing · column, strategy, value? · Fill missing cells: constant (needs value), mean/median (numeric columns), forward (previous non-missing). coerce-type · column, to · Coerce to number/date (ISO)/boolean; failures become missing and are counted in the log. normalize-unit · column, factors · Strip a unit suffix and multiply ({"万": 10000, "亿": 100000000}); plain numerics convert too. trim · columns? · Trim whitespace of string cells (all columns when omitted). map-values · column, map, else? · Exact-match mapping; unmapped values stay (keep, default) or become missing.

The source file is never overwritten. With outputPath the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only. With dryRun: true no file is written and nothing is persisted — the result returns the per-column cleaning plan and the expected contract/diffPreview. The result also carries a pre-delivery contract summary (dedupe before/after rows, uniqueness over the dedupe key or full rows, non-null regression for fill-missing columns, type regression for coerce-type columns, and the per-column decision trace), and a clean-diff before/after profile report is persisted to the storage domain.

data_report({ key?, kind?, format? })

Reads persisted reports back from the storage domain. Pass key (the exact reportKey a prior run returned) to fetch one report, or kind (profile/clean/clean-diff/verify/citations) to list every report of that kind chronologically; exactly one of key/kind is required. Malformed or missing keys fail loud. format: html (with key) renders the report as a self-contained offline HTML document — inline CSS/JS, no external requests, the DAMA six-dimension scorecard, and the profile/cleaning summary tables (profile/clean reports only).

data_verify({ path, rules, expectations? })

Evaluates verification rules. Rule reference:

Rule · Extra fields · Semantics not-null · column · Fail missing cells (null/empty/whitespace). unique · columns · Fail every row whose key combination repeats (missing participates). range · column, min?, max? · Fail missing/unparseable cells and values outside the inclusive bounds (at least one bound required). regex · column, pattern, flags? · Fail missing or non-matching cells (full JS regex). enum · column, values · Fail cells whose trimmed text is not listed. cross-column · left, op, rightColumn?, value? · Compare per row: numeric when both sides parse, dates compare as epochs, strings only for ==/!= (exactly one of rightColumn/value). freshness · column, maxAgeDays, asOf? · Fail dates older than maxAgeDays before asOf (default: now); unparseable/missing fails.

expectations reconciles deterministic metrics against expected values: rowCount, columnSum, columnMean, uniqueCount, nullCount (each with column except rowCount, plus expected and an optional relative tolerance in [0, 1]). Each expectation yields passed plus actual/expected/tolerance; a mismatch is a normal passed: false verdict, never a tool error. Invalid metrics, missing columns, and out-of-range tolerances fail loud.

A missing cell fails every rule that reads it. Evidence is capped at evidenceRowLimit failing rows per rule.

ctx.dataQuality (for other plugins)

const result = await ctx.dataQuality.verifyCitations({
  dataset: 'holdings.csv',          // resolved against workspaceRoot
  citations: [
    { id: 'c1', path: 'rows[3].nav', value: 1.234, tolerance: 0.01 },
    { id: 'c2', path: 'summary.annualReturn', value: '12.34%' },
  ],
})
// result.results[i] = { id, status: 'verified' | 'mismatch' | 'not-found' | 'unverifiable', actual?, note? }

Locators walk the dataset document: CSV/TSV load as { columns, rows } (so rows[3].nav resolves), JSON is the parsed value, JSONL the array of parsed lines. Numbers compare with relative tolerance (|a-b| <= tolerance * max(|a|, |b|)); a CSV string cell that parses numerically compares as a number; strings compare exactly; incomparable type pairs are unverifiable. The service also exposes profileDataset / cleanDataset / verifyDataset (the same operations the tools call).

Permissions & data

• Reads workspace dataset files (allowlisted extensions only).
• Writes only: the data_clean output file (explicit outputPath, workspace-confined, never the input) and reports in the data_quality storage domain under the harness data directory.
• No network, no credentials, no external processes — all parsing and statistics are in-process TypeScript.
• Reports may contain sample cell values from your datasets (bounded by evidenceRowLimit and display truncation); the session log records tool arguments and results as usual.

Security boundaries

• Path confinement — dataset and output paths must resolve inside the session workspace (verifyCitations uses workspaceRoot); .. escapes and outside absolute paths reject, and both sides are normalized before comparison (Windows slash-safe).
• Bounded work — maxRows / maxFileSizeMB guards reject oversized inputs loudly; abort signals cancel long loads mid-stream.
• No overwrite — data_clean refuses an outputPath equal to the input path.
• Deterministic computation — same input, same output; the only clock is the one injected for freshness defaults and report timestamps.

Known limitations

• Session events are adaptive. The published 0.1.7-rc.2 line (like the earlier rc lines) has no plugin session-event registration surface and its Session.append cannot stamp the ignorable marker, so appending an unknown data-quality/* type would make the session log unreadable on restore. The plugin therefore appends only when the host knows the vocabulary or supports the ignorable append flag; on the published line the storage-domain report is the durable record.
• CSV dialect — comma/tab with RFC-4180 quoting, header row required, blank lines skipped, no delimiter auto-detection or comment lines.
• Type parsing is strict — numbers have no thousands separators; dates are YYYY-MM-DD / YYYY/MM/DD / ISO-like datetimes (UTC); booleans are true/false/yes/no/1/0. Everything else profiles as string/mixed — clean it with coerce-type when intended.
• JSON must be tabular for the tools (array of flat objects); verifyCitations walks arbitrary JSON documents.
• No ML anomaly detection, no PII masking, no databases, no SQL — rule-based suspicion notes only.

Development

pnpm install
pnpm run typecheck && pnpm run typecheck:ci && pnpm run typecheck:checkout && pnpm test && pnpm run build
pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:readme-sync && pnpm pack