Benchmarks: recovering feature boundaries

Can a planner recover a codebase's real feature boundaries? We flatten feature-modular repos into type folders, then score each method against the original folders: MoJoFM (0–100), ARI, c2c_cvg and the cluster-count ratio. TurboMQ needs no ground truth: it scores the import graph alone (many imports inside a module, few between modules), not whether the modules are the features. Every entry gets both MoJoFM and TurboMQ, each with its own best deterministic method. Each scramble is one commit by our own executor, then checked by our own certifier as a pure move: ghostfolio, hoppscotch-H1 and hoppscotch pass (static checks); kaneo (failing K3c, K4) fails (static checks). Each entry below shows its verdict; a scramble is scored whatever its verdict, since the ground truth is its original folders. Deterministic baselines are always shown next to the planner, because classic methods often win (ICSA 2026 found they beat LLM decompositions by 62% on MoJoFM, docs/plan.md §7; our G1 gate agrees on kaneo, docs/gates.md). Real type-sliced targets have no complete ground truth: they get TurboMQ and back-edges, and where a target's own history labels some of its files (a partial answer key), MoJoFM and the rest over those files only.

In these results the planner beats the best deterministic method on MoJoFM on 3 of 5 entries (ghostfolio, hoppscotch-H1 and hoppscotch), and loses on 2 (immich and kaneo); on TurboMQ it beats it on 1 of 5 entries (ghostfolio), and loses on 4 (hoppscotch-H1, hoppscotch, immich and kaneo). Each measure has its own best deterministic method, and immich's MoJoFM is over its 2024 key. Our own validator rejects the plans of ghostfolio (banned-name: "app" is a junk drawer or a type word, not a feature) and kaneo (banned-name: "middleware" is a junk drawer or a type word, not a feature). ac run refuses such a plan; the bench scores it as it came back. Counting only the plans the validator accepts, the planner wins on MoJoFM on 2 of 5 (hoppscotch-H1 and hoppscotch) and on TurboMQ on 0 of 5.

Bold: the higher of the best deterministic method and the planner on each measure.

summary.md · each entry's JSON is linked under its table.

Nemotron 3.5 Super VL as the planner

We measured NVIDIA Nemotron 3.5 Super VL as the planner in place of Nemotron 3 Ultra: the same pipeline, prompts and entries, 3 runs per entry for each configuration, with Super VL's thinking off, and on with room to finish (256k-token answers). Ultra was recorded afresh for this comparison; the overview's planner values above are one earlier draw per scramble, and immich's is the demo's recorded plan (planner rules 1). Each cell is the mean over the runs, min to max in parentheses, every run as delivered; bold is the highest on each row.

Each configuration's results: ultra, super-vl-direct, super-vl-256k (summary.md; each entry's JSON is next to it).

With thinking on and room to finish (256k-token answers), Super VL's mean MoJoFM is above Ultra's on all 4 scrambles (a permutation test over the runs, stratified by entry, gives p ≈ 0.0007), at 2.5 times Ultra's cost and 6.4 times its time per plan. On TurboMQ none of the 45 planner runs score above the best deterministic method, and on immich (TurboMQ 10.26 against Ultra's 10.20, MoJoFM over its 2024 key 88.37 against Ultra's 89.15) we see no measurable difference between the two. The demo keeps Ultra. Among the reasons: the recorded demo run reproduces byte for byte with the frozen tool, and switching would mean new code and new recordings after the feature freeze; 3 runs per entry are few; in 11 of its 30 calls Super VL's long reasoning names the public project it is planning (ghostfolio, kaneo or immich), and in most of them tries to recall its structure; and Ultra was compared at medium_effort only, never at full thinking.

The full write-up is bench/experiments/s8b-super-vl/README.md in the repository.

Entry by entry

ghostfolio scramble, default

Target apps/api/src/app @ 347cd06a40, 154 movable files.

Scrambled by our own executor in commit 9e65a3a639: 156 scored files moved into type folders (a test next to its source moves with it and isn't scored), 3 declined (they stay), 219 excluded; certificate ✓PASS (static).

Ground truth: 153 feature files in 34 original folders; 54 hard files (their name doesn't give their feature away); 16 of the movable files give their feature away in their path (0 renamed after it, 0 still in a feature folder, 16 under a sub-folder their feature kept; 28 more that no method sees are not counted).

Planner: 42 groups (38 modules), from the seeds draft (export names), plan sha256 9048c87d30c7, $0.0878 when recorded, 1 of 2 calls replayed from cassettes/s7-bench.jsonl and 1 recorded into it ($0.0274 spent by this run).

• 6 feature files are pinned in the facts (no method can place them) and not scored.
• The validator rejects the planner's plan (banned-name: "app" is a junk drawer or a type word, not a feature); it is scored as it came back.
• The planner starts from the seeds draft (export names): plannerDraft takes the export names only when they leave fewer files contested than the file names, and here they leave 4 against 5 (no ground truth is read).
• The planner's plan moves 7 of 154 files out of their draft group and merges 3 draft groups into others (44 draft groups, 42 plan groups). The winning edit, ultra#2, had 44 group entries and 4 moves (4 name their file by its id, not its path); the validator found 3 problems with it (banned-name 3) and dropped or repaired each before applying it. Planner rules 2 read a file by the id the prompt showed for it. The critic made 28 suggestions and its gate kept 12.
• Leakage, over the 154 movable files every method sees: 16 give their feature away in their path (0 still in a feature folder; 0 leaky renames; 16 in a sub-folder that only files of their feature share). Sub-folders kept under the type folders: calculator (6 files, 1 feature), errors (2 files, 2 features), ghostfolio (7 files, 1 feature), queue (3 files, 1 feature); the planner's prompt shows them, and no deterministic method groups by them. 28 more leaky files are pinned or not read by the facts, so no method sees them: not counted.

ghostfolio.json

hoppscotch-H1 scramble, H1

Target packages/hoppscotch-backend/src @ 63273f8e59, 186 movable files.

Scrambled by our own executor in commit bf37a0f639: 156 scored files moved into type folders (a test next to its source moves with it and isn't scored), 1 declined (they stay), 22 excluded; certificate ✓PASS (static).

Ground truth: 156 feature files in 25 original folders; 46 hard files (their name doesn't give their feature away); 0 of the movable files give their feature away in their path (0 renamed after it, 0 still in a feature folder, 0 under a sub-folder their feature kept; 3 more that no method sees are not counted).

Planner: 37 groups (35 modules), from the seeds draft (export names), plan sha256 e38cd9aeaa64, $0.1275 when recorded, 1 of 2 calls replayed from cassettes/s7-bench.jsonl and 1 recorded into it ($0.0191 spent by this run).

• 1 feature file is pinned in the facts (no method can place it) and not scored.
• The planner starts from the seeds draft (export names): plannerDraft takes the export names only when they leave fewer files contested than the file names, and here they leave 12 against 85 (no ground truth is read).
• The planner's plan moves 5 of 186 files out of their draft group and merges 8 draft groups into others (45 draft groups, 37 plan groups). The winning edit, ultra#1, had 45 group entries and 17 moves (17 name their file by its id, not its path); the validator found 4 problems with it (banned-name 2, duplicate-move 1, foreign-name 1) and dropped or repaired each before applying it. Planner rules 2 read a file by the id the prompt showed for it. The critic made 9 suggestions and its gate kept 2.
• Leakage, over the 186 movable files every method sees: 0 give their feature away in their path (0 still in a feature folder; 0 leaky renames; 0 in a sub-folder that only files of their feature share). Sub-folders kept under the type folders: constants (1 file, 1 feature); the planner's prompt shows them, and no deterministic method groups by them. 3 more leaky files are pinned or not read by the facts, so no method sees them: not counted.

hoppscotch-H1.json

hoppscotch scramble, default

Target packages/hoppscotch-backend/src @ 63273f8e59, 186 movable files.

Scrambled by our own executor in commit a38bd84671: 156 scored files moved into type folders (a test next to its source moves with it and isn't scored), 1 declined (they stay), 22 excluded; certificate ✓PASS (static).

Ground truth: 156 feature files in 25 original folders; 46 hard files (their name doesn't give their feature away); 0 of the movable files give their feature away in their path (0 renamed after it, 0 still in a feature folder, 0 under a sub-folder their feature kept; 3 more that no method sees are not counted).

Planner: 29 groups (22 modules), from the seeds draft (file names), plan sha256 615def2cbbab, $0.1375 when recorded, 2 calls replayed from cassettes/s7-bench.jsonl ($0.0000 spent by this run).

• 1 feature file is pinned in the facts (no method can place it) and not scored.
• The planner starts from the seeds draft (file names): plannerDraft takes the export names only when they leave fewer files contested than the file names, and here they leave 12 against 10 (no ground truth is read).
• The planner's plan moves 7 of 186 files out of their draft group and merges 14 draft groups into others (43 draft groups, 29 plan groups). The winning edit, ultra#1, had 43 group entries and 7 moves; the validator found 2 problems with it (foreign-name 2) and dropped or repaired each before applying it. The critic made 9 suggestions and its gate kept 8.
• Leakage, over the 186 movable files every method sees: 0 give their feature away in their path (0 still in a feature folder; 0 leaky renames; 0 in a sub-folder that only files of their feature share). Sub-folders kept under the type folders: constants (1 file, 1 feature); the planner's prompt shows them, and no deterministic method groups by them. 3 more leaky files are pinned or not read by the facts, so no method sees them: not counted.

hoppscotch.json

immich real

Target server/src @ 02063972e8, 269 movable files.

A real type-sliced target, with no complete ground truth. MoJoFM, ARI, c2c_cvg, the cluster ratio and MoJoFM hard are over its 2024 key, immich's March 2024 domain-layer feature folders followed through git history to the pinned commit: 47 of the 269 movable files in 23 features, mostly in src/services/ (23) and src/dtos/ (22) with 2 elsewhere; 41 of the 47 are named after their feature (key bench/ground-truth/immich-2024.rsf, details in bench/ground-truth/README.md in the repository). Every method places all 269, and only those 47 count. TurboMQ (higher is more modular) and back-edges (lower is better) cover every file.

Planner: 59 groups (38 modules), from the seeds draft (file names), plan sha256 b4c7c548883c, $0.1998 when recorded, 2 calls replayed from cassettes/s6-immich.jsonl ($0.0000 spent by this run).

• The planner starts from the seeds draft (file names): plannerDraft takes the export names only when they leave fewer files contested than the file names, and here they leave 31 against 29 (no ground truth is read).
• The planner's plan moves 9 of 269 files out of their draft group and merges 23 draft groups into others (81 draft groups, 59 plan groups). The winning edit, ultra#3, had 81 group entries and 29 moves; the validator found 4 problems with it (foreign-name 3, kind-conflict 1) and dropped or repaired each before applying it. The critic made 12 suggestions and its gate kept 6.

immich.json

kaneo scramble, default

Target apps/api/src @ b80818fe2e, 454 movable files.

Scrambled by our own executor in commit aad6fc97e3: 400 scored files moved into type folders (a test next to its source moves with it and isn't scored), 1 declined (they stay), 12 excluded; certificate ✕FAIL (static, failing K3c, K4).

Ground truth: 401 feature files in 47 original folders; 234 hard files (their name doesn't give their feature away); 54 of the movable files give their feature away in their path (53 renamed after it, 1 still in a feature folder, 0 under a sub-folder their feature kept; 4 more that no method sees are not counted).

Planner: 64 groups (55 modules), from the seeds draft (file names), plan sha256 bcb39422b864, $0.1714 when recorded, 1 of 2 calls replayed from cassettes/s7-bench.jsonl and 1 recorded into it ($0.0649 spent by this run).

• The validator rejects the planner's plan (banned-name: "middleware" is a junk drawer or a type word, not a feature); it is scored as it came back.
• The planner starts from the seeds draft (file names): plannerDraft takes the export names only when they leave fewer files contested than the file names, and here they leave 125 against 29 (no ground truth is read).
• The planner's plan moves 22 of 454 files out of their draft group and merges 50 draft groups into others (114 draft groups, 64 plan groups). The winning edit, ultra#2, had 114 group entries and 29 moves (29 name their file by its id, not its path); the validator found 3 problems with it (bad-name 2, banned-name 1) and dropped or repaired each before applying it. Planner rules 2 read a file by the id the prompt showed for it. The critic made 16 suggestions and its gate kept 10.
• Leakage, over the 454 movable files every method sees: 54 give their feature away in their path (1 still in a feature folder; 53 leaky renames; 0 in a sub-folder that only files of their feature share). 4 more leaky files are pinned or not read by the facts, so no method sees them: not counted.

kaneo.json