dsh-model-arena

dsh-model-arena

Run the same prompt through every model you have, and see the difference without squinting.

Install

dsh plugin --profile web add dsh-model-arena

Or from source:

dsh plugin --profile web add github:hj01857655/dsh-model-arena

What it does

• Run a prompt through N models. Same prompt, identical parameters, outputs collected side by side.
• Compare. Line-level diff between any two model outputs — additions, deletions, and unchanged lines, with a summary of how many lines changed.
• Score. Auto-scorer (length, latency, reference match) + manual user ratings (1–5). Scores are persisted per run, so you can rate a model later and the leaderboard updates.
• Leaderboard. Aggregate scores across runs into a ranked table — win rate, average score, and total runs per model.
• Benchmark suites. Save a named set of prompts + models, re-run after changes, get a regression report that tells you which models improved and which got worse.
• Panel. Settings page listing recent runs with drill-down to comparison view, leaderboard, and regression reports.

CLI

dsh-model-arena list                              # list recent runs
dsh-model-arena compare <run> <modelA> <modelB>   # diff two models in a run
dsh-model-arena rate <run> --model M --rating 4   # rate a model output

Panel

A Model arena page inside Settings. The host registers GET /api/model-arena.panel and rebuilds the payload from the store on every request. A headless host simply skips the route.

The panel shows:

• Recent runs with model outputs side by side
• Line-level diff viewer for any pair of models
• Auto-scores and user ratings per model output
• Leaderboard across all runs
• Regression reports for benchmark suites

Layout

Develop

npm ci
npm run typecheck
npm run build
npm test

License

MIT