breakthrough-harness

breakthrough-harness

Make your research agent hard to fool — starting with itself.

中文版 · works with any agent stack · pure methodology + two runnable demos

That is python examples/toy_loop.py (< 30 s, numpy only): a candidate that secretly fits the calibration answers looks like a breakthrough, and the held-out confirmation executes it. This half screen is the entire philosophy of the repository.

Most agent harnesses teach an agent how to work. This one teaches a research agent how not to deceive itself — because in research the failure mode is rarely "the code crashed" and almost always "the number looked great and was wrong." Every rule here was paid for by a real failure; none is hypothetical. The repository follows its own rules: what these pages claim, tests/ checks on every push.

The core claim

Breakthroughs are a throughput problem. breakthrough ≈ many cheap attempts × a scoring function that is hard to fool.

Serial, hand-crafted experiments produce a few dozen attempts per year; the systems that actually produced constructive breakthroughs (program search over mathematical constructions, tournament-style hypothesis engines) share one skeleton: parallel cheap generation + automated selection that is hard to fool. You cannot copy their compute; you can copy the skeleton — if the scoring function is engineered with the rigor a referee would apply. That engineering is this repository.

And a quieter corollary: when every attempt is expensive, the rational move is to audit what exists rather than build what might fail — audits have guaranteed deliverables. Teams that keep sliding into negative-result papers are responding rationally to the price of an attempt. Fix the price and the sliding stops.

What is inside

The other demo: the flat sweep that lied

Act 1  lam in {0.1, 1, 10, 100}   -> scores 2.1, -0.0, -0.0, -0.0   "flat, the parameter is inert"
Act 2  one gradient-norm evaluation -> every value crushed the data term; balance is ~0.05
Act 3  sweep around the balance    -> clean interior peak at lam = 0.0015, 22.6 dB

python examples/force_balance.py: a four-decade hyperparameter sweep that looks flat because the data term is a mean and the regulariser a sum — every tested value sat on one side of the force balance. One gradient-norm evaluation finds the real range. Rule 3, live.

Use it with your agent, whatever it is

The methodology is plain markdown — nothing here depends on any vendor. The adapters just package it for wherever your agent reads instructions:

All adapters carry the same methodology; the five plain-markdown ones are generated from a single source (adapters/_core.md) and CI fails if they drift. If your tool reads AGENTS.md, install either that or the tool-specific file — not both — and the same goes for the skill and a rules file: one route per tool, or the rules arrive twice.

Quick start

git clone https://github.com/GuoCheng24/breakthrough-harness
cd breakthrough-harness
python examples/toy_loop.py     # < 30 s, numpy only

The five habits, in one screen

1. Null models first. Before believing any score, ask what a method that learned nothing would score. If your pipeline can't tell them apart, fix the pipeline before running a single experiment.
2. Calibration and evaluation never touch. Tune on one set of files, report on another. A gain that does not survive the held-out set does not exist.
3. A baseline you cannot beat is a recipe you have not finished reading. Published baselines hide layers: the optimizer, the loss, the metric convention, the operator. Read until your reproduction matches.
4. Claims have a polarity. The main sentence of a result must be constructive — "we propose X, it solves Y, the number is Z." Audit output supports the claim; it is never the claim.
5. Every guard must be shown to fail. A check never seen firing on a deliberately broken input is decoration — guards in this very repository were caught silently passing before this rule existed.

What this is, and is not

It plugs into whatever you already run. It replaces nothing.

Where the skeleton comes from

The core claim is not folklore; the named systems and known failure modes:

• FunSearch — Romera-Paredes et al., Mathematical discoveries from program search with large language models, Nature 625, 468–475 (2024), doi:10.1038/s41586-023-06924-6: program search + automated evaluator producing new constructions (cap set, bin packing).
• AI Co-Scientist — Gottweis et al., Accelerating scientific discovery with Co-Scientist, Google, arXiv:2502.18864 (2025): tournament-style hypothesis generation/ranking with automated review.
• The reusable holdout — Dwork et al., Science 349, 636–638 (2015), doi:10.1126/science.aaa9375: why repeated consultation of a held-out set silently turns it into a calibration set (the reason loop/ gives held-out data a lifecycle and a budget).
• Test-set overfitting in practice — Recht et al., Do ImageNet classifiers generalize to ImageNet?, ICML 2019, PMLR 97: distribution of gains that fail to transfer, measured at field scale.
• Deep learning tuning playbook — Godbole et al., 2023, google-research/tuning_playbook: the discipline of scientific vs. nuisance hyperparameters this repo's sweep rules descend from.

Other things from the same desk

• groundwork — a research pipeline for coding agents whose first stage can return NO-GO; on PyPI as groundwork-research
• doubleblind — re-derive every number in a README or paper from committed files, and brief a reviewer that was told nothing
• batch-logprob-gap — in low-precision training the same token gets a different log probability depending on its batch shape — measured, with what it does and does not do to GRPO

More at github.com/GuoCheng24.

License

MIT. Use it, fork it, disagree with it.