DSH-AutoKnit

AutoKnit

English | 简体中文

Throw in a PRD — it decomposes tasks, dispatches agents, verifies acceptance, and splits recursively on its own. Cheap to build, cheaper to maintain, zero babysitting.

AutoKnit is an open-source divide-and-conquer execution framework that sits between "chatting with your agent step by step gets exhausting" and "enterprise-grade heavy frameworks are too expensive". It never burns context playing "general coordinator": the program does all scheduling (0 tokens), LLMs only contribute intelligence. You review the plan once, and merge the code once at the end.

Benchmarked: on a ~7,000-line task, AutoKnit cold start was the cheapest (749K tokens), fastest (37 minutes), and had the highest test density (26.4/1k lines) — 19% cheaper than a single interactive agent and 41% cheaper than lh-harness. Full comparison in Section 2.

Rerun (2026-09-02, n=2 — repeatability confirmed): the same engine on the same task closed clean at 701,977 tokens / 33.6 min / 97.2% cache hit — 6.3% cheaper and 9% faster than the 748,802 baseline above, with 0 rework rounds, 0 executor swaps, 0 human escalations across 6 rounds and zero manual intervention. The 08-30/31 three-way table is deliberately kept at its original accounting so the −19% / −41% ratios stay internally consistent; treat 700–750K as the honest range for a task of this size.

The execution substrate is dsh (the DeepSeek harness) — it owns sessions and model calls; AutoKnit owns decomposition, dispatch and acceptance. Comparison baseline lh-harness (LongHorizon-Harness by AMAP-ML, MIT, paper) is an excellent long-horizon harness — fresh-context execution, durable verified state, independent auditing, recoverable progress — and it also runs codex as executor. Some numbers in this README come from fair head-to-head runs against it; the comparison is about cost accounting at scale, not about which framework is better designed.

---

1. What problem does it solve

Pain point: Vibe Coding developers spend endless time and energy and still don't know how to arrange and architect things.

AutoKnit fills exactly this gap: engineering and contracts replace "repeated conversation", so LLMs only contribute intelligence. Target scale: 1,000–10,000 lines — you save not only on code production itself, but on the hidden costs of back-and-forth wrangling during development and repeated rework during maintenance.

---

2. Benchmarked (2026-08-30/31, same machine, same accounting)

Detailed accounting notes at the end of this section; per-case full reports in docs/benchmark.md. These are initial reference directions (single-run samples). You're welcome to share your own measurements in Issues — the benchmark will keep updating with community feedback. Subjects: AutoKnit (deepseek-v4-flash @official API, thinking low, explicitly reproducible), lh-harness (same model and tier, thinking low), single interactive agent (GLM-5.3-Flash with thinking, single-session continuous mode; its billing excludes independent audit). Billing = uncached input + output (each tool counts tokens differently; normalized). Every token number in this document is billed accounting and comparable across tables — the single exception is the "modification experiment" table (raw, incl. cache), which is for relative comparison between the four rows only.

Module-level tasks (500–1,000 lines, three independent implementations, same PRD)

m01 DSH task-state data bridge (~1,000 lines expected, 5 acceptance items)

m02 session/usage data bridge (5 acceptance items; delivery lines = business source lines, same accounting)

m02 is the final data from the 09-01 framework (same PRD, same model deepseek-v4-flash + thinking low). AutoKnit 303,973 — 16% cheaper than lh (363,531), 40% cheaper than the interactive agent (506,763); delivery of 2,322 lines is 24% thicker than lh; 82 tests all green (auditor verifies item by item).

m03 human-reply service (smallest module, ~600 lines, 5 acceptance items; same PRD, three independent implementations, all independently re-verified)

Same PRD for all three (sha256 identical); the difference is the execution model itself: AutoKnit's auditor collects evidence item by item plus programmatic checks, lh's auditor "quits when acceptance passes", the interactive agent has no independent audit. m03 is the smallest module (~600 lines): AutoKnit 107,786 — 1.5× lh, 3.0× interactive. At the 300–500 line scale orchestration overhead can't amortize; the interactive agent is the best tool (consistent with the sweet spot below).

m04 create-agent/create-session workspace binding (~750 lines, 5 acceptance items; AutoKnit as a single module — different decomposition granularities of the same task differ 4.6× in cost: granularity itself is the biggest cost lever)

m04 is 09-01 framework data (single module, done in 1 round, 139s). AutoKnit 62,685 — 55% cheaper than lh, only 22% more expensive than the interactive agent (51,513) — whose billing excludes independent audit.

Large task (~7,000 lines: plan-only mode + programmatic code merge + DSH panel plugin; all three passed acceptance)

Modification experiment: four deliverables given the same change request (add --dry-run to merge)

Real scenario: you ask an agent to change a feature; it first figures out "which files to touch", then starts. If its estimate is far off, it misses files — at best multiple rounds of rework, at worst it ships with hidden problems. So we tested: gave four independently generated deliverables (two AutoKnit builds / lh / interactive) the same change request, recorded the "predicted blast radius" with codegraph first, then measured the actual changes.

⚠️ Accounting note: this table is the only raw (incl. cache) accounting in the document — during the modification experiment that interactive tool only recorded cache-inclusive totals and couldn't split them. The four rows share one accounting, so relative comparison is valid; absolute values cannot be converted against the other tables (billed accounting).

Four takeaways from the data

One-line summary: AutoKnit buys more maintainable, more reliable code for fewer tokens — the savings don't come from making the LLM work less, they come from turning orchestration and acceptance into 0-token programs so every token is spent on real output.

1. Small tasks (≤1,000 lines): interactive tools are still cheapest (36–52K vs AutoKnit's 62–304K) — that's why our sweet spot starts at 1,000 lines: single-session, zero orchestration overhead, it's simply optimal. But once a task has a cross-module dependency chain (like m02's three modules), AutoKnit overtakes (303K vs 507K interactive, 40% cheaper).
2. Large tasks flip the order: at 7,000 lines AutoKnit cold start was cheapest (749K, -19% vs interactive, -41% vs lh), fastest (37min vs 4.5h), highest test density (26.4/1k lines). The interactive single-session model inflates context at scale (861K of uncached input, all spent re-reading new content).
3. The interactive agent's cheapness excludes audit: its billing has no independent acceptance. Auditing three interactive deliverables at equivalent strength cost us 559K afterwards (~190K each) — the interactive agent's true cost = generation + audit. AutoKnit's auditor is built in; the quote includes verification.
4. Modification blast radius is predictable and quotable: an agent's inherent flow when changing code is "read the structure → predict which files to touch → change them → run tests". Because AutoKnit's deliverables are decomposed + modularized + contract-bounded, what the agent reads is a clean module topology — measured prediction deviation ≤±1, maximally avoiding the three rework accidents: incomplete changes, one-touch-ripples-to-many, and shipping problems from half-finished edits. Test density of 26.4/1k lines is 3× the baselines — the same lines of code, with over 3× the test escort.

Divide and conquer without reinventing wheels: UCD upstream digest (0 tokens)

The classic cost of module-level divide-and-conquer: each module runs in its own clean session and doesn't know what the others already built — wheels get reinvented. AutoKnit solves this with UCD (upstream capability digest): once an upstream module passes acceptance, the program (AST extraction, 0 tokens) generates an UPSTREAM.md — public interfaces, function signatures, one-line purposes — injected into the downstream executor's startup context. Downstream modules import and reuse directly, without reading upstream source (contracts + digest suffice; coupling stays low) — freeing capacity for robust code, tests, and documentation.

v4 rerun (2026-09-03, fixed engine + UCD): same PRD, same baseline — 701,977 billed (-6.3% vs baseline), 33.6 min, 95 test cases, 26.5% doc density (docstrings counted). The isolation dividend of divide-and-conquer and the reuse dividend of a monolith, at the same time.

Sweet spot (honest boundaries)

---

3. How it works (architecture)

                 PRD
                  │
        ┌────────▼────────┐
        │     planner      │  Splits by coupling: tightly-coupled code goes together
        └────────┬────────┘   + first-task detailed checklist + inter-module contracts
                 ▼
        ┌────────▼────────┐
        │     executor     │  Independent session, responsible for its module only
        └────────┬────────┘
                 ▼
        ┌────────▼────────┐
        │     auditor      │  Verifies the acceptance list item by item + programmatic evidence
        └────────┬────────┘
                  │ Remaining modules still large?
                  ▼ large (threshold tunable, ≈1000 lines) → split recursively
                    small → the executor finishes the remainder itself
                  ▼
             All done ✅

1. planner splits by coupling: tightly-coupled code lands in the same module (if changing A means changing B, they belong together); modules communicate only through data contracts + interface contracts and never read each other's source.
2. executor runs + warm continuation: independent session per module; when the remaining volume drops below the threshold it doesn't spawn a new block — the current executor finishes the remainder — its context is still warm, saving tokens and keeping quality coherent.
3. auditor acceptance: independent role, verifies the acceptance list item by item + programmatic evidence collection (pytest / semgrep / boundary checks); failures get sent back.
4. split recursion: only splits further when the remainder exceeds the threshold; each block is "swallowed in one bite".
5. Granularity is the biggest cost lever and the quality/overhead balance point: different granularities of the same task differ 4.6× in cost (63K–336K). Our balance threshold is ~1,000 lines per executor task — above it, split recursively (smaller context per block); below it, let the current executor finish (warm context saves tokens).

Rejection and escalation (the boundary of "fully automatic" — when you actually show up)

An auditor rejection ≠ starting over. Repairs are always in-place: module artifacts stay, the executor's next round continues with its 【previous-round feedback】 (done / to-do); even when the executor is swapped, the replacement takes over existing progress (progress snapshot + handover bundle) — never from scratch.

The escalation chain after a rejection (default config, fully automatic — you only show up if everything fails):

auditor rejection
  → same executor fixes in place (up to 2 rounds, REVIEW pinpoints the to-dos)
  → a different executor takes over (once)
  → recursive split: break the module into smaller pieces
  → model upgrade fallback (flash → pro)
  → only then needs_human, and a human takes over

Two exceptions escalate immediately without burning rounds: failures rooted in upstream/contract (retrying is pointless — fix the dependency first), or environment problems (rate limits / network — bounded backoff, 3 retries, since swapping executors is useless).

The dashboard's "pending decisions" contains only two kinds — neither is a code-level micro-decision (code-level items are already verified by the auditor's item-by-item evidence):

• Human acceptance items: GUI appearance / real-world scenarios / experience — dimensions a framework cannot verify. Code is all green; these are listed for you and an external AI to review.
• Modules needing a decision: modules where automation is exhausted, with root cause attached (delivery / environment / contract / stall) and options A abandon / B change approach / C pause / D custom.

What a contract looks like

Modules share no code — only shapes. Each module's task book carries contract files (auto-generated; you can also declare them explicitly in the PRD and the planner will respect that):

# contracts/m01-task-state.yaml (example)
interface:
  dsh.task.list:
    direction: F→R
    returns:
      tasks: "List[TaskSummary]"   # sorted by urgency desc
      task:
        id: str
        stage: planning|executor|auditor|rejected|reassigned|needs_human|done
        modules: "List[{id: str, stage: str, rejected_count: int}]"
data:
  snapshot.json:        # upstream file this module reads read-only
    run_id: str
    phase: str
  dispatch.jsonl:       # event stream, append-only
    events: "List[{seq: int, type: str, payload: dict}]"
boundary:
  may_read: [contracts/, shared/]
  may_write: [modules/m01/]

The executor's entire world is this contract plus its own module directory — it doesn't need, and is not allowed, to know the global picture.

Why it stays high-quality while saving tokens

AutoKnit's secret to saving tokens isn't "making the LLM work less" — it's "letting the LLM do only what it's best at": writing code. All orchestration, verification, dependency and reuse guidance is done by the program in the 0-token scheduling layer, so a modest amount of LLM intelligence produces higher-quality work.

• 0-token programmatic scheduling: planner/split/runner are programs, not a "general coordinator" LLM. In the large-task run AutoKnit's orchestration cost was literally 0 — every token went to real output.
• Cache friendly: frozen-prefix discipline + independent sessions per module; cache hit rates of 94–98%.
• A deterministic workstation: contracts, interfaces and boundaries are pre-processed by the program into the task book — 100% of executor tool calls go to writing code, zero to exploration.
• Upstream capability digests (UCD): when a module finishes, the program AST-extracts its public interfaces and injects them 0-token into the downstream executor's context — downstream reuses upstream capabilities directly (m03 reusing m02's summary), no rewriting, no exploration (measured 32% cheaper for that module).
• Programmatic auditor evidence: pytest/semgrep/boundary checks are pre-run by the program into an evidence layer; the auditor only spot-checks, never reads everything — audit cost share dropped from 34% to 22–28%, still item-by-item.
• Dependencies installed once: python_packages declared in the task are aggregated and installed by the bootstrap (120s timeout fallback) — executors never install on the fly or reinvent wheels.
• Cheap models + low thinking tier work: deepseek-v4-flash (flash tier, thinking low) produced thick deliveries end to end — the architecture takes "arranging and architecting" off the LLM's shoulders.

---

4. How to use

Flow

Today's agents can already "ask clarifying questions and turn a vague requirement into a complete PRD" — that upfront conversation usually takes 3 rounds. Hand it to AutoKnit from there; what you save is hours of back-and-forth and the information loss of a bloating context:

1. Tell your agent the requirement        ← you only bring the idea
2. Your agent asks clarifications, produces a complete PRD  ← usually 3 rounds
3. autoknit plan-only <task dir>   ← review the plan: how many modules, how many lines each, contract list (no execution, no cost)
4. autoknit run                    ← split / dispatch / write / verify / continue / recurse — fully automatic
5. Get modules/ module code + merge notes
6. Your agent merges against the notes, one round  ← you review once

Dependencies (from zero)

1. Python 3.11+, git, npm
2. pip install autoknit — installs the autoknit command in one shot (framework bundled; incl. fw-protocol / fw-scaffold / the data bridge). Source route: git clone https://github.com/Renjie-hub-byte/DSH-AutoKnit.git && cd DSH-AutoKnit && bash install.sh
3. Install [dsh](docs/quickstart.md appendix A) (the DeepSeek harness, npm install -g @deepseek-ai/dsh) and log in, or point ~/.autoknit/config.yml at your dsh path and credentials
4. autoknit doctor — one-shot health check: dsh binary / credentials / model routing / panel connectivity, with human-readable fix instructions for anything missing

Commands & flags

About merging

autoknit merge is pure program, zero LLM mechanical merging: it produces ① the directory skeleton positioned by dependency topology, ② each module's target interface files, ③ cross-module import wiring, ④ four conflict lists (name clashes / naming mismatches / interface signature drift / needs-semantic-merge — each marked "needs human decision"). Measured: hand the merge notes to an agent and one session finishes the merge; the semantic-level fusions are clearly listed.

---

5. Failure modes and protections (all bitten and fixed — so you don't have to)