DSH-EditApart

EditApart — AI taste-driven editor (agent preset)

EditApart is an automation-faithful editor for the EditTogether community. It takes raw footage, builds a shot inventory, proposes an EDIT DECISION SCHEMA against a declared editorial rubric, renders it deterministically with ffmpeg/scenedetect (and ImageMagick for photos), critiques the render against the rubric, and revises until it converges.

It does not operate a video-editing GUI. It reasons about editorial judgment. The plugin registers both video and photo tools plus the taste-model tools (15 total) under one identity; the engines are bin/edit_apart_core.py (video), bin/photo_core.py (photo) and bin/taste_model.py (the per-creator model).

The core design

• Schema is the interlingua. The same EDIT DECISION SCHEMA is the thing you train on, the thing you render from, and the thing you critique. No gap between "AI intent" and "the render."
• General grammar is shared; taste is per-creator. One shared style-brain (film-grammar parameters) + a small per-creator identity latent z_u in a per-creator GGUF. Only the identity is per-creator — the artifact holds z_u and a digest of the trunk it belongs to, nothing else.
• Reward is dense + group-relative (GRPO). The critic scores a render AND yields per-element keep/drop/why deltas; r = overall + λ·Σ(deltas). Group relative advantage A_k = (r_k − mean(r))/std(r), then a PPO-clip step.
• Muon on weights, AdamW on the latent. Muon orthogonalizes momentum for 2D weight matrices; never the identity latent (Newton-Schulz degenerates on 1D). AdamW handles z_u and all 1D/bias/embed params.
• Self-play interleaved with history. propose_schema emits a group of candidate schemas from a deterministic grid; the group is logged, one candidate is selected, and the render/vision/creator outcome is logged back against it. Training consumes those groups overnight.
• Three model portions, and only one is the session model. General (the session model: reads footage, proposes, critiques) · FilmGPT style-brain (shared film grammar) · creator taste (z_u). A video-in/out model like GLM-5.3 (Z.ai / zai, OpenAI-compatible) replaces the general portion only — it reads the raw clip via {"type":"video_url","video_url":{"url":…}} instead of pre-extracted frames; the FilmGPT and taste portions are untouched.
• Model-authored video is opt-in architecture, not a preset claim. ffmpeg renders the final cut (render_schema). Fully AI-generated clips come from a separate store plugin (or the harness video_generate tool, e.g. grok-imagine-video), installed opt-in. When present, you may generate a clip and insert it into a regular edit as a generated segment (source + AI clip are separate inputs the renderer concatenates). The preset itself never claims to generate video.
• Photo editing is image-only — one fewer modality. The same taste-driven loop edits a single image via the edit-apart plugin (bin/photo_core.py), which needs only image input (no video modality, no video-in/out registration), so it works with the existing image-capable general model. The ImageMagick renderer is resolved at runtime (DSH_IMAGEMAGICK/PATH) and is not a hard dependency — a web-profile in-browser renderer may supply it.

The pipeline

The editor's tools (mounted as a preset plugin at plugins/edit-apart.mjs) are the thin facade; bin/edit_apart_core.py (video) and bin/photo_core.py (photo) are the deterministic engines they shell out to.

source ─► inventory ─► shot inventory
               │
     features ─► per-shot luminance/motion/audio
               │
  propose_schema + rubric ─► GROUP of candidate EDIT DECISION SCHEMAS
               │                    │         (logged to .editapart/taste_samples.jsonl)
               │                    ▼
               │        taste identity scores the group ─► selected schema
               │
    render_schema ─► deterministic ffmpeg render
               │
    critic_edit ─► score + per-element keep/drop/why deltas
               │                    │         (logged as that candidate's reward;
               │                    │          chosen_by=creator|agent = revealed preference)
    revise_schema ─► revised schema ─► re-render (bounded loop)
               │
   train_identity ─► shared style-brain GGUF + per-creator z_u GGUF

Status of the taste model. The identity learning is implemented and wired into the loop, not a scaffold. propose_schema builds and logs a group of candidates, selects one with the creator identity when it exists, critic_edit logs the reward for the candidate that was actually rendered, and train_identity fits the shared style-brain plus that creator's latent z_u and writes real GGUF artifacts. The measured claims and their limits are in Taste model below — including what it does not do yet (trained selector, not a fine-tuned language model; video only).

Photo editing (image-only)

A parallel single-image loop, mounted via the edit-apart plugin: photo_inspect → photo_propose (rubric → photo EDIT DECISION SCHEMA) → photo_render (ImageMagick) → photo_critic (objective + subjective) → photo_revise. It needs one fewer modality than video (image, not video), and its renderer is not a hard dependency (resolved at runtime; a web-profile in-browser ImageMagick/ffmpeg may supply it).

The taste model is wired in here too, with its own photo/v1 layout, its own group log and its own style-brain (style_brain_photo_v1.gguf), so a creator ends up with one identity per modality. Three things make this modality harder than video, and all three are measured rather than assumed:

• No temporal axis to lean on, so the layout synthesises spatial structure: crop geometry plus statistics of the region the crop keeps, read from a 4×4 region grid that photo_inspect computes from the same downscale as the global stats (the global numbers are unchanged).
• Candidate scoring costs a render. The photo reward is defined on the rendered image, so a K-candidate group is rendered and scored K times (video's objective critic is pure arithmetic). group=1 with no dataset stays the legacy, render-free proposal.
• A quantised reward ties the group. Exposure was a threshold, so a 6-candidate group scored 6 × −0.05, spread 0.0 — exactly zero group-relative signal. The reward is now graded (full credit inside the tolerance, linear falloff outside), which took that group to spread 0.47 and left 11/18 training groups informative.

Measured on a synthetic 1600×1200 image, two creators with opposite framing taste (tightest vs loosest crop) sharing one frozen trunk: 10/10 held-out briefs from the training family are picked differently, 10/10 in the predicted direction (mean crop area 0.155 vs 0.389). On one unseen crop geometry it transfers too — 10/10 over ten distinct briefs — but with a smaller margin: mean crop area 0.30 vs 0.45, a gap of 0.15 against 0.233 in-distribution. That is the geometry/content confound of docs/paper-findings.md (Finding 4c) narrowing the effect, not erasing it. (An earlier "0/10, no transfer" reading came from the objective bug described under Corrections; it was an artifact and is withdrawn.)

A four-arm follow-up (tests/experiment_photo_transfer.py: {1 image, 3 images} × {fixed, varied crop family}, shared probe ladder, matched steps) found the repair is partial and along the content axis, not the geometry axis: only the 3-image/narrow-family arm transferred to an unseen image and geometry (3/3, ratio 0.400), while both varied-geometry arms gave 0/3. A no-taste control identity showed the ordering metric's noise floor is ±0.36 — larger than most effects measured — so treat these results as suggestive, not established. docs/paper-findings.md Findings 4c/4d has the tables, the power analysis and the rules (train on the family you will use; include an out-of-family check and a no-taste control).

Taste model (wired, measured)

bin/taste_model.py is the per-creator model; bin/train_identity.py is its CLI. The loop in bin/edit_apart_core.py drives it.

Architecture. A shared trunk over 16 named edit features (pacing, shot selection, tempo, energy, order) feeds a hidden layer that the creator latent modulates three ways:

u = tanh-free ranking score =
      (a ⊙ (1 + tanh(W_s z + b_s))) @ W2 + b2     # latent gates the hidden activations
    + (z @ W_z + b_z)                              # latent output offset
    + x · (z @ W_l)                                # per-creator linear readout over the features

z_u is the only per-creator tensor, so a creator artifact is ~1 KB while the trunk is shared. The map is nonlinear in z (tanh gates, multiplicative terms) and interpolatable (the latent is continuous; a midpoint latent scores between its endpoints — asserted in the test suite).

Objective. For each clip the loop logs a group of candidates with the critic's dense reward. GRPO turns that into group-relative advantages A_k = (r_k − mean r)/std r, and the update is a PPO-clip surrogate on the group-softmax selection policy, plus an auxiliary pairwise preference loss over the pick that was actually rendered and a small entropy bonus — but only when the pick came from an external judge (chosen_by=creator|agent|human). A choice the system itself made (taste: the identity's selection was rendered; critic: nobody picked, the objective critic only scored) trains neither the preference term nor the reward bonus. It still learns through the group-relative advantage over the critic's rewards, which measures the edit rather than claiming anything about the creator. The loop therefore records who picked, not just what was picked; taste_status reports how many groups were excluded for this reason. (If the group record and the later reward record disagree, the reward record — the engine's own flow — wins.) Utilities are standardised within the candidate group before the softmax: this keeps the logits responsive so the latent can always reorder candidates — with an unbounded-then-squashed output the utilities saturate under the margin objective and the latent loses all authority over the ordering (measured: 12/12 neutral briefs identical, versus 12/12 different after the fix).

Optimizer split. Muon (quintic Newton-Schulz, coefficients and nesterov form from the reference Muon implementation) on 2D weights; AdamW on z_u and every 1D/bias parameter. Muon's lr is in spectral-norm units, so on these small matrices it must be ~10× below an Adam lr — at the higher lr the model stops fitting (measured: training accuracy stalls at ~0.5 versus ~0.85).

Artifacts. Real GGUF v3 files (F32 tensors + metadata): a shared style_brain_video_v1.gguf and a tiny per-creator <creator>.gguf carrying z_u, the feature layout, and the digest of the trunk it was trained against. A digest mismatch is refused rather than silently scoring with the wrong trunk.

How the loop feeds it

# 1. propose a GROUP (logged automatically); the best candidate by the identity
#    is selected, or by the objective critic reward when no identity exists yet
propose_schema inventory=<inv> rubric=<rub> group=4

# 2. render the selected candidate, then log that group's outcome
critic_edit schema=<schema> inventory=<inv> rubric=<rub> group_id=<id> \
            candidate=<idx> chosen_by=creator

# 3. inspect and train
taste_status
train_identity creator=erkin            # writes .editapart/identity.gguf
train_identity creator=erkin freeze_style=true   # later: only z_u moves

# the photo loop is the same protocol on its own log/identity
photo_propose src=<img> inspect=<insp> rubric=<rub> group=4
photo_critic schema=<schema> rubric=<rub> result=<render> group_id=<id> \
             candidate=<idx> chosen_by=agent
taste_status modality=photo
train_identity modality=photo creator=erkin

chosen_by=creator|agent matters: the pick is logged as a revealed preference, becomes that group's top reward, and gives the trainer a preference term. A rubric-derived reward alone cannot identify per-user taste: with no user-dependent term in the objective, two creators with opposite tastes differ on only 4/12 neutral briefs (p ≈ 0.21 against the unrelated-picks null), versus 8/12 (p ≈ 0.0006) once the revealed pick is in the objective (margins +3.6 s vs +7.0…+9.4 s). The override additionally fits the creator's own corpus far better than the auxiliary loss (0.83/1.00 vs 0.22/0.26), so both mechanisms are kept; tests/experiment_revealed_preference.py reproduces the whole table.

Multi-input effects (the v2 layout)

The v1 feature layout describes the resulting cut sequence — shot counts and durations, cuts per minute, pace band, motion, coverage, order, luminance, audio RMS. A transition, a grade, ducking and an overlay are decisions about how inputs combine (a transition joins two clips, ducking relates speech to music, an overlay composites layers), and none of them appears in that list. So with v1 the identity could only choose which selection to use; it could not set an effect's parameters at all. That is a real limitation, not a detail: an NLE-shaped editor is mostly effects.

video/v2 adds ten effect-parameter features (transition fraction, duration, alignment and asymmetry; grade deviation; LUT; reframe; duck depth; segment gain; overlay fraction) under a new spec id — never an edit of v1, because the layout list is part of an artifact's digest, so a v1 identity keeps loading and keeps refusing a v2 trunk instead of being silently reinterpreted with shifted columns. A log may hold both; the majority layout trains and the rest are counted (groups_other_spec), since mixing vector layouts would either crash the first forward pass or mis-align columns.

Two halves are needed, and both exist: a group that spans effects, so a preference can be learned — propose_schema(… effect_grid=true) pairs the selection grid with effect knobs (the default grid varies selection only, so every candidate in a legacy group agrees on every effect feature and nothing could ever teach an effect preference) — and an identity that can choose them: propose_schema(… refine=N) runs a deterministic coordinate ascent over the effect axes (transition type/duration/alignment, ducking, grade strength) scored by the identity, and the tuned schema joins the group as one more candidate. Two properties are deliberate:

• It needs an effect-aware identity. A v1 trunk has 16 inputs and cannot score a 26-dim vector, so the request is declined with a reason rather than silently ignored — and the feature spec a group is logged under is explicit (--feature-spec), because a group logged under v1 can never teach an effect.
• No identity means no effect taste. refine_effects with no scorer returns the schema untouched: with no identity there is no taste to apply, and maximising the objective critic there would quietly turn a rubric rule into a taste claim.

Measured (tests/test_taste_model.py::TestEffectTaste): groups are logged under v2 with a reward that rises only with the transition duration — a hidden preference expressed solely through an effect parameter — an identity is trained on it (train_rank_acc ≥ 0.8), and the refiner then chooses a dissolve and pushes the duration past 0.8 s from an all-cuts seed. The control run with no identity changes nothing. TestEffectFeatures pins the blindness of v1, the superset relation and the mixed-spec rule.

They are now rendered, and the rest are refused. cmd_render executes transitions — xfade between runs, acrossfade for the audio, and an outgoing tpad extension so all three alignments are exact — which is what makes the transition features true: the rendered duration equals the planned one to the frame, and the picture at the boundary differs from a hard cut. grade.lut renders too: the repository ships generated .cube LUTs (luts/warm.cube, luts/cool.cube, reproducible with tools/make_luts.py --check), a bare name resolves to one of them and a name that resolves to nothing is refused. The all-cut path is byte-identical to the renderer that produced every earlier measurement, so nothing had to be re-measured. The fields the renderer still does not execute — audio.duck (no music bed is mixed), overlay (no overlay media), transform.position (the output canvas is the source frame) — are declared-but-unrendered and refused by assert_renderable at propose, refine and render time, with a reason. That makes the schema's own contract ("every field in the schema is renderable by the renderer") an enforced property instead of a comment: an edit decision can no longer describe a picture the renderer will not produce.

Per-effect credit

Credit is attributed to the decision that actually varied, in two forms:

# the attribution table for one group: what changed, why, and how much it weighed
python bin/edit_apart_core.py attribute <dataset.jsonl> --group-id <gid>
# the corpus-wide profile: what the rewards (and an identity) prefer, effect by effect
python bin/edit_apart_core.py attribute <dataset.jsonl> [--identity ID] [--style S]

Three properties make the table trustworthy rather than plausible. Rows are keyed by shot, not by index — candidates of one group can select different numbers of shots, so index 6 of one candidate is a different shot from index 6 of another and an index-keyed diff invents changes (a real value against None) and blames them on an effect. The axis comes from the logged knobs, so a coupled decision counts once: transition type/duration/alignment are one decision (a duration is meaningless without the type) and they legitimately move together, where a slot count would misreport them as three decisions. And each row carries the critic's own judgement for that shot (decision/why/reward_delta, now logged on the reward record) beside the group-relative advantage, so a row states its basis and its magnitude.

attributable is the flag to check before believing any of it: exactly one knob axis engaged AND the candidates selected the same shots. A group that varies several decisions, or that changed its material as well as its effects, has no single-decision credit to give — the profile counts and skips those groups rather than folding an unattributable number into a claim. That is why --effect-grid holds the selection fixed and varies only the effects; exploring both dimensions is two proposals, not one. (Nothing measured depended on the previous pairing, which was introduced and replaced inside this unreleased sequence.)

One finding this makes visible: in an effect group the objective critic's reward is flat across transition values — it scores duration, motion and target, and cannot see a dissolve at all. So the effect columns of the profile discriminate only through the identity (the taste_mean column), which is precisely why an effect preference has to be learned from an external pick rather than from the rubric. The test test_the_critic_cannot_see_a_transition fails if that ever stops being true.

Not yet done, and worth naming: credit is still per candidate, not per effect — the group-relative advantage is still broadcast over the whole candidate rather than graded into the per-row table — the table records the attribution (which decision changed, on what basis, with what weight) but the gradient is not yet distributed over its rows, so a group that varies several decisions still trains one advantage for all of them (the segment-local-credit design in docs/expansion-plan.md). The effect axes are a fixed grid rather than continuous optimisation, and effects carry no provenance of their own beyond the pick provenance described above.

The training datum (and why the raws are kept)

A taste datum is not a timeline with a score; it is a group: alternatives produced under one fixed request, materials and constraint set. Schema v3 of the group record keeps everything a replay needs, in the record itself:

Before v3 a record kept only derived numbers — no rubric, no inventory, no timeline, no source identity — so a defect in the feature extractor or a change to the critic made the corpus un-remeasurable. This project's own corrections log is a history of needing exactly that: three conclusions flipped after defects and each had to be re-measured on the same data. Keeping the raws is the precondition for that discipline, and an NLE makes it sharper, because there the edit is a reference structure over a media pool rather than a schema over one file.

# re-derive every logged group from its own record and compare
python bin/edit_apart_core.py replay <dataset.jsonl> [--limit N]

It recomputes each candidate's feature vector and the objective reward from the recorded timeline + inventory + rubric, verifies the hashes, and reports older records as not replayable rather than guessing. The subjective leg is a person's or agent's judgment: it is reported as not recomputable and never re-invented. tests/test_taste_model.py::TestLogReplay is the acceptance test, with two negative controls — a corrupted stored vector and a changed source file are both caught.

Why annotations stay on our side. Measured: through the FCPXML adapter nothing of ours is even written — not the effect, not the marker, not the metadata (tools/otio_conformance.py documents the lossless container). So the ledger carries the judgment (why, decisions, provenance) keyed by content_hash, the timeline carries structure, and export is one-way and lossy: anything crossing an editor boundary returns as a new revision with a new judgment, never as recovered metadata.

Not yet v3: the photo engine still writes pre-v3 records (a replay reports them as not replayable), and media retention is a policy here rather than an enforced garbage-collection rule.

Measured results