Mulligan dataset release plan

Inventory checked 2026-09-21; release copies verified 2026-09-22. All 246 existing HF repositories and two simulation evaluation bundles are public in mulligan. Per-repository receipts record pinned source revisions, payload checks, episode inventories and preserved Git history. The episode ancestry audit covers all 153 derived views, including six reconstructed by exact trajectory matching.

Release catalog · Policy Arena · Execution record.

The public Arena selects the mainline Marker D2, Square D2, Routing D2, Square-Narrow and Square-Broad results and their required training ancestry. The broader HF inventory also preserves explicitly named same-campaign ablation views. Standalone D1/pilot campaigns and unrelated simulations are excluded.

The canonical results decision selects the August 21 Square D2 R5 contemporaneous trio. The manuscript snapshots, tracker analyses, internal Arena classification and release export use that selection. Historical July studies remain explicitly labeled archives.

The full old-to-new mapping is mapping.csv. manifest.json adds commit pins, parent relationships, camera availability, licenses, file-inventory fingerprints, evidence, and repository-specific release checks. training-recipes.json translates all 46 locked simulation DIVL input lists and four declared final real-world critic train/eval lists. out-of-scope.csv accounts for the other 837 repositories returned by the source-namespace audit.

Organization

The organization is mulligan, display name Mulligan. It was created on 2026-09-22. The dated namespace probes in namespace-checks.json preserve the pre-registration availability audit.

Inventory and release boundary

The release groups these into 110 mainline repositories, 103 supporting repositories, and 33 same-campaign ablation views. The ablation views include no-CF alternatives, simulation Sobol alternatives, and Broad R0 first100. They share recordings with mainline views and belong in a separate collection on the organization page. Keeping them reproduces the existing mainline comparisons without admitting unrelated experiments.

The audit examined all 1,116 publicly listed ankile dataset repositories and pinned the metadata/file trees of 279 campaign candidates. It excludes 33 candidates with explicit reasons in the mapping. Six excluded candidates lack meta/info.json. The remaining 837 source repositories are listed separately. This is an inventory of that namespace and the cited experiment contracts, not a claim about unlisted data in other accounts or local-only recordings.

These counts come from pinned meta/info.json, with known split repositories removed using lineage or the cited split configurations. They count recorded episodes, including repeated initial states, and do not claim trajectory-content deduplication. Hours are recorded frames divided by each dataset's FPS, not operator time. The current trees to copy total 220.22 GB, including derived views; this is neither git-history size nor a storage billing estimate.

Names

All destination IDs use lowercase kebab case. Domain and task come first. Indexes have two digits so lexicographic order matches numeric order. Historical dates, seeds, checkpoints, quota details, and sampler parameters belong in the card and release metadata.

Task tokens are exactly real-marker-d2, real-square-d2, real-routing-d2, sim-square-narrow, and sim-square-broad. DAgger variants are baseline, mulligan, mining-no-cf, sobol-free-cf, and sobol-no-cf. R0 treatment demos use sobol, since no counterfactual collection occurred at R0. validation is the held-out teleop view, not an autonomous policy evaluation. The two Broad preview views retain the explicit first100 suffix.

Robot policy-view method tokens are baseline-dp, sobol-dp, mulligan-dp, mulligan-idql, and mining-no-cf-dp. Simulation autonomous comparator names retain auto-bc-n1-shared, auto-filtered-bc-n1, auto-plain-il-n1, auto-iql-n32, and auto-iql-success-bc-n32. Their collection policy was historical scalar IQL even where the current evaluated critic is DIVL. All autonomous rollout dataset names end in policy-rollouts; the source names ending in diagnostic-rollouts are normalized.

cNN identifies a collection increment. rNN identifies an evaluated model round. bNN identifies a distinct recorded evaluation session/block, never a dataset split. Block numbers in the committed mapping are fixed release identities. If a later block is added, append a number; do not rerun the original ordering over a changed inventory and silently renumber existing blocks.

Routing needs two round axes

Do not merge those source repositories or relabel their collector policies. The earlier collections used the velocity lineage; C09 used UMI-relative collectors. The shared final evaluation contains 15 policies and 750 episodes in one original repository. It stays one repository with a policy/round lookup table. Source: Routing D2 lineage and executed analysis configuration.

Nine historical Routing evaluation parents and their 22 policy views are included as dependencies. The final critic input script uses 14 Ours rollout views for training and eight baseline views for evaluation only. Their IDs use legacy-rNN to retain their original evaluated model round. This is the required Routing D2 training ancestry, not a release of the retired Routing D1 campaign as a separate benchmark.

Evaluation identity and reuse

Marker R2 needs three source evaluation blocks; Square R3 needs three. The current R5 headline uses ankile/real01b-md2-r5-repeat-base-dp-filmtiidk4-c200k-n32-s2026070704 for Marker and the fresh-seed s2026081801 redo for Square. Earlier R5 blocks stay supporting because their policy views exist in the campaign. Source: the per-block summary paths in the manifest and line configurations.

An evaluation recording can later become production evidence for another critic. The Marker R5 input script directly includes its earlier DP-versus-IQL evaluation parent. Label a block's training eligibility per consuming model, rather than promising global train/test separation from its name. Preserve manifest_idx, seed, visit/session identity, policy IDs, and original episode IDs. Same seed alone does not make separate sessions the same physical reset. Keep HU/Oracle diagnostic starts separate from production sampler fit.

Simulation evaluation bundles

Create mulligan/sim-square-narrow-r00-r03-eval and mulligan/sim-square-broad-r00-r03-eval. These are result datasets built from existing artifacts, not existing LeRobot repositories to duplicate. evaluation-bundles.json enumerates their cells, locked grids, source file patterns, and separate probe-campaign roots. The production rollout datasets in the copy map are not substitutes for held-out grids.

The authoritative inventory is full_protocol.yaml, with 46 cells and five seeds per cell. Include its dataset revision pins, cell/arm/filter definitions, original actor artifacts, current DIVL artifacts, state manifests, and manifest hashes. The result ledger names all 230 seed results. training-recipes.json maps every declared HF input.

For each cell/seed, package the exact files consumed by the ingestion code: the current locked-grid result or all required shards, plus the pinned parent comparison where released. Probe cells use the earlier probe campaign paths handled by that ingestion code. Add canonical N=1 and iterative-IL comparator records from grid.csv and the paper-baseline registry; mark scalar-IQL comparison results as historical. Archive the locked 8,000-state Narrow and 30,000-state Broad grids with their seeds and hashes. The mainline publication manifest supplies verified public artifact locations for the released aggregate outputs.

This audit verified the protocol, source paths, and aggregate result ledger, but did not hydrate and inspect every simulation grid shard. Before packaging, resolve each raw artifact through the S3 consumer, verify the grid hash and expected point count, and enumerate every missing file. Never substitute a CSV success rate for missing per-rollout records while calling the bundle full trajectories. The paper simulation evidence record explicitly documents missing historical speed/throughput episode records. Publish those as aggregate-only evidence if they cannot be recovered. Do not regenerate evaluations silently and present new outcomes as original recordings.

Copy and verify

1. Register the organization, grant the publishing account write access, and fix the namespace in the manifest. Create an organization card and five HF Collections, one per task, with links to mainline, supporting, and ablation data.
2. Freeze the selected source SHAs. Every selected source currently has main == v3.0; all 52 distinct HF inputs pinned by the full simulation protocol match the audited current main. Recheck immediately before each copy. Execution uses the pinned cross-repository file-copy API and transfers source Git history to source-main. Destination main has its own release metadata commit. Require the source head and source payload fingerprint to match the audit before and after copying; compare payload identities rather than expecting source/destination main commit IDs to match. Copy into private staging repositories first.
3. Copy the 93 original repositories, then the 153 derived repositories in parent-first order. Leave ankile sources intact. Use exist_ok=False; an existing destination requires a content comparison, not automatic acceptance. Record source SHA, destination SHA, file inventory, tree bytes, and operation time per row. A pilot copy of one small state dataset and one real camera dataset must pass before batching the remaining copies. API behavior has been checked in installed source and official documentation; simulation and camera pilots passed on 2026-09-22; see the execution ledger. The duplicate endpoint was limited to 15 repositories/day, so the copier uses the documented file-copy API plus Git history transfer.
4. Verify the unmodified destination tree against the pinned source using every path, byte size, Git blob ID, and LFS SHA-256. Verify historical source commits and required tags remain accessible. Then make one reviewed metadata-only release commit per repository. Archive original lineage under meta/source_dataset_lineage.json before normalizing current identity fields. Remap repo_id, parent_repo_id, derived_repos, derived_repo_ids, mutually_exclusive_with, and repository prefixes in view_family_id using the complete map. Keep original task/round/policy identity, embedded SHA-pinned manifests, episode IDs, and source references as provenance; add release aliases in meta/release.json. Do not rename observation columns, transform actions, trim episodes, relabel outcomes, or re-encode video as part of copying.
5. For the 54 selected repositories without lineage JSON, build explicit release lineage from the cited split scripts and available split sidecars. Sim R0/R1 need particular care. Check every derived episode against its original parent; keep no-CF/free-CF mutual exclusion and CF donor links. Fail on an unresolved parent or an episode mapping gap. Counts alone cannot establish this mapping.
6. Standardize dataset cards. Include task and reset distribution, domain, collection/model round axes, data role, sampler, collection policy and consumer policies, action convention, camera keys, FPS, episode/frame totals, label revision/review coverage, license, source SHA, parent/child links, and train/validation/test use per model. The selected cards declare apache-2.0; preserve that declaration. The repository's LeRobot push API defaults to that license, so the card string alone is not an independent release-rights review. Document the 25 two-camera real repositories accurately. Route serial-name mappings through verified station metadata if encountered; do not invent camera mappings.
7. Preserve Routing's frozen 4a7ea324753a6723dd83a9f8f0da91dd52dbe2d0 label snapshot. It has 502 human-reviewed episodes plus 248 eval-time labels according to its committed provenance. It also remains the audited current main. Any subsequent review gets a new version and a separate result table. Distinguish machine stage predictions from reviewed labels.
8. Run a fresh-cache LeRobot read at the destination SHA. Validate active episode metadata references, episode/frame counts, source flags, CF links, outcome/timing fields, action/proprio schemas, and video frame alignment. Decode representative frames and inspect episodes from each recording schema and collection/eval family. Check payload hashes again after metadata edits so the data/video bytes remain unchanged. This audit did not download or visually inspect trajectory/video payloads; that is an execution gate.
9. Point destination v3.0 at the release commit after metadata edits, and create a separate immutable release tag. Publish exact source-to-release SHA pairs. The v3.0 tag is loader compatibility, not the release version. Training/eval examples should use the immutable release revision. Keep the source repos and historical script references intact; update new release configs and the website through the mapping.
10. Package and verify both simulation evaluation bundles. Render the catalog from the final manifest, enable destination links only for verified public repositories, then make the staged datasets public and expose the task Collections. Check each public file anonymously, open the page, and test its task/role/round filters and parent links. If verification fails, keep that destination private and report it; never delete or redirect the source as rollback.

Copying is recommended because historical experiment scripts and caches already identify the source namespace. HF also supports transfers with redirects, documented in repository settings, but this plan makes independent release copies with explicit lineage.

Overview page

The generated page uses this manifest directly. It starts with the five task totals and a collection-to-evaluation explanation, then provides search and filters for task, role, tier, and model round. Every row shows its proposed release name, linked pinned source, episode count, parent, and any outstanding release checks. Proposed destination names stay plain text until publication is verified. Excluded candidates remain inspectable behind a filter.

The page is a planning catalog, not a claim that the release is live. Its generated HTML and downloadable manifests can be hosted on S3 for review and later linked from the paper website and HF organization card.

Reproduce the plan

Run from the repository root. Raw audit caches are disposable; the committed mapping is the review record. Refreshing against mutable source main produces a new proposed plan and requires a diff before adopting changes.

uv run python experiments/2026-08-27/01_audit_real_hf_datasets.py --author ankile --output /tmp/mulligan-release-audit/all_info.json --workers 8
uv run python experiments/2026-09-21/03_audit_mulligan_release.py --inventory /tmp/mulligan-release-audit/all_info.json --cache /tmp/mulligan-release-audit/details --workers 4
uv run python experiments/2026-09-21/04_build_mulligan_release_plan.py --audit /tmp/mulligan-release-audit/details/candidates.json --inventory /tmp/mulligan-release-audit/all_info.json --namespace-checks docs/mulligan-release/namespace-checks.json --output docs/mulligan-release
uv run python experiments/2026-09-21/05_render_mulligan_release.py --manifest docs/mulligan-release/manifest.json --output /tmp/mulligan-release-preview/index.html

The namespace-check file records the dated observations; these reproduction commands do not recheck or reserve the name. Keep the HF source data unchanged until the plan's copy, lineage, and payload-verification steps are executed.