CVDP RTL Agent: what passes the hidden testbench

How it works

The graph is read_task → plan → edit → check → reflect → (edit | finalize). The LLM writes the plan, the edits and the decision. Everything else is code: the file tool writes only under rtl/, verif/ and docs/, and the checks are real tools (Icarus iverilog -g2012, verilator --lint-only -Wall, a Yosys read and synth, and a testbench the agent writes itself). The replay below is a long problem from the same run: nine turns, ending in a submit after the agent's own testbench had failed.

Turns 1 to 8 are drawn from the problem's totals (each one ended in edit_again, because another turn followed); their individual events are not in the committed excerpt. Turn 9 and the hidden-test line are quoted from it.

The sweep, as a schematic

Stop rules, checked in code before every step

• Turns: 10 edit, check and reflect cycles by default.
• Dollars: $0.10 per problem, provider-reported cost only. A call whose cost is not reported stops the problem (cost_unreported), because the budget can no longer be enforced.
• Wall time: 480 s, under the harness's 600 s agent timeout. Each LLM call has a hard deadline 2 s before the cap.
• On any stop or crash, the last edit that compiled is kept, the summary is written, and the agent exits 0.

Milestones, M0 to M3d

M2: three models, 92 problems, two runners

Each model ran all 92 problems once on an x86-64 and once on an Arm64 runner, with the same image and settings, against the null agent's floor of zero.

The capture: every problem, every full run

Below, each row is one problem and each column one full sweep of all 92, like the channels of a logic analyser. A filled cell is a pass; a bare baseline is a fail. The six M2 sweeps come first, then the 13 new M3 sweeps (M3's run A1 is M2's Luna Arm64 sweep, so it is not repeated). Rows are sorted by how often the problem passed.

• Run-to-run noise is large. Between its x86-64 and Arm64 runs, Haiku changed result on 10 problems, Luna on 9 and DeepSeek on 12. The direction is not consistent, so this reads as the model's sampling, not the runner.
• Haiku against Luna is a tie. Counting each problem's passes over both runners, Haiku is ahead on 12 problems and Luna on 12. Both beat DeepSeek (24 against 6, and 21 against 6).
• 29 problems passed in all six M2 sweeps and 28 in none. 24 failed in every one of the 20 full runs of M2 and M3.
• 3 seeds on a 30-problem subset vary by up to 4 problems for one model (seeds table).

M3: six changes to the agent, none detectable

Each arm is an ablation: one setting changed, everything else held. Every arm uses gpt-6-luna (the cheapest per solved problem in M2), all 92 problems, Arm64 and 4 threads, and runs twice. Each arm is compared with the base (A) problem by problem: a problem's rate in an arm is its passes over the arm's two runs, so 0, ½ or 1. The effect is the mean of the differences, with a paired bootstrap interval and an exact sign test.

Intervals are from tools/ablate.py as committed in docs/reports/m3/arms_paired.md; the point effects and better / worse counts are recomputed from the CSVs by site/data/derive.py and agree.

F (20 turns) is the only arm worth a second look, ahead in both runs (62.0% and 58.7% against 57.6% and 55.4%). But only 6 of its 184 problem-runs used more than 10 turns, and 1 of those passed. The extra turns account for at most 1 of its 7 extra passes; the rest is noise.

What the agent did with its own testbench

M3b: a richer harness, or a memory of lessons?

M3b holds the model (gpt-6-luna), the Arm64 runner and every M2 default fixed, and changes two things in a 2×2 design: the harness that drives the model (M2's hand-built LangGraph graph, or Deep Agents), and whether the agent is given a frozen lessons memory. Each arm ran 3 times on the 86 problems of the evaluation set: the 92 minus 6 that no agent can reach, by a rule fixed before any scored run. Arm A is a fresh run of M2's agent on the same image, so all four arms share an image and a period.

The paired effects, null findings first

Intervals are tools/ablate.py's paired problem bootstrap as committed in docs/reports/m3b/; the point effects, the better / worse / same counts and the exact sign tests are recomputed from the run CSVs by site/data/derive.py and agree.

• The lessons memory did not help. C − A is +1.6 [−3.1, +6.6]: 10 / 8 / 68 problems better / worse / the same, sign test p = 0.82. C's three runs solved 51, 51, 51 of 86. On Deep Agents, D − B is +3.1 [−0.4, +7.0], p = 0.15: not detectable either.
• The harness may help a little. B − A is +4.3 [+0.0, +8.5], p = 0.21: the interval's lower end rounds to zero and the sign test is far from significant. On top of the memory the harness is clearer: D − C is +5.8 [+1.9, +10.1], 9 / 1 / 76, p = 0.022, the one comparison whose interval excludes zero.
• Why that was a hypothesis, not yet a finding. With four pre-registered comparisons, a Bonferroni threshold of 0.0125 on the sign tests passes none of them, and the bootstrap intervals are not adjusted. D − A (+7.4 [+2.3, +12.8]) was not pre-registered, so it is exploratory only. For scale: on these 86 problems the noise floor is −2.3 [−9.3, +3.5], and M3's largest effect (20 turns) is +4.1 [−0.6, +9.9]. So M3b named one hypothesis, the harness gain B − A, and a separate run was fixed to test it.

The confirming run: B − A on fresh data

The test was written down and committed before any of its runs was dispatched (pre-registration). It is a single pre-registered comparison, B − A, so there is no multiple-comparison correction. It uses 6 new runs per arm (seeds 4 to 9) with M3b's settings unchanged, and M3b's own runs are not pooled in, so the data that suggested the hypothesis does not also test it. The decision rule: if the paired 95% interval excludes zero and the effect is positive, the effect is confirmed; if it excludes zero and is negative, reversed; otherwise not confirmed, with the size it rules out.

• How strong. 16 / 7 / 63 problems better / worse / the same; the exact sign test gives p = 0.093. The sign test was not part of the rule. It is shown so that the evidence reads as moderate, not overwhelming.
• The size repeats. Pooled pass@1 is A 57.9% (299/516) and B 62.6% (323/516), close to M3b's +4.3.
• So does the mechanism. Deep Agents leaves fewer failures untested (tests_failed_no_selfcheck: A 121, B 79) and stops on its round cap far more often (A 28, B 101 problem-runs), at about the same cost per solved problem (≥ $0.0126 against ≥ $0.0119; one call in each arm had no reported cost).
• What it covers. One model (gpt-6-luna), one problem set (the 86 free agentic problems) and one Deep Agents version. The memory arms C and D, and D − C, were not re-run: the memory finding stands as M3b reported it.

Why Deep Agents differs: fewer untested failures, more round-cap stops

Deep Agents runs a free tool-call loop: the model calls compile, lint, synthesis and its own testbench as tools, as often as it likes, up to a cap of 22 model calls (the most any M2 problem made in 10 turns). It leaves far fewer failures untested (tests_failed_no_selfcheck: A 58, B 39, D 33), but stops more often on its round cap (failures labelled stopped_by_cap: A 11, B 29, D 33). A cap stop still submits the best edit kept so far, and some pass: B hit the cap 49 times and 20 of those passed. With the lessons, the LangGraph agent has more failures whose own testbench passed (C 21 against A 15); with both, the fewest (D 8).

What the memory learned

The builder read 87 attempts by arm A on 29 heavy-split problems (none in the evaluation set; 19 of the attempts passed), asked the model for general lessons in batches, and froze 40 of them into snapshot b783931cbf160595 for $0.0031. At run time the agent gets the top 5 by BM25 over its task text, with no model in the retrieval. The lessons are generic process advice, several near-duplicates, and 6 are never retrieved for any of the 86 problems (L07, L08, L17, L28, L30, L38: about compile diagnostics and turn budgets, words the problem statements do not use). The four retrieved most often, quoted as built:

The interface lessons did not move the interface failures: compile_fail_hidden stays at 12–13 and signal_missing_hidden at 2–3 on every arm. A null C − A says that this memory does not help, not that no memory could.

Cost

All four arms cost about the same: $0.0067 to $0.0074 per problem-run, and $0.0124 (A), $0.0119 (B), $0.0113 (C) and $0.0106 (D) per solved problem, with no unreported call in any of the 1,032 problem-runs. The arms differ in pass rate, not in price. The scored sweeps cost $7.27 in all, the learning runs at least $3.41.

M3c: two model-free checks, as fixed steps

M3 and M3b found the failures in two places: interface mismatches with the hidden testbench, and weak self-checks (no testbench of its own, or one that passes on code the hidden tests fail). M3c adds one deterministic check for each, with no model in it, to the LangGraph agent, and runs each against a fresh base A, 3 runs an arm on the same 86 problems. M − A and I − A are the two pre-registered comparisons.

• M, a mutation-score gate before submit. Yosys mutate makes 12 mutated copies of the agent's design, and the agent's last testbench must fail on at least half of those that compile. With no testbench, the gate refuses the submit. After 2 gate failures the agent may submit anyway.
• I, an interface diff by elaboration after every edit. pyslang elaborates the design, and its modules, ports and parameters are compared with the interface the agent was given. It never reads the hidden testbench. A mismatch turns one submit into one more edit, then lets it through.

The gate changes the kind of failure, not the number

The gate did what it was built to do: it will not let the agent submit without a testbench, so failures with no testbench of the agent's own fell from 56 on A to 0 on M. But they did not become passes. Failures where the agent's own testbench passed and the hidden tests still failed rose from 14 to 50, and stops on a cap from 4 to 19, as the gate's extra rounds used up turns. Requiring a testbench is not requiring a correct one. The interface check moved neither of its own labels: the two interface labels together are 16 failures on A and 16 on I.

What the checks found

The gate scores the agent's testbench, not the design. A testbench that checks the wrong behaviour thoroughly kills mutants and passes the gate, and the hidden tests then fail the design. The interface check reads the given interface from prompts and documents by rules, not by a parser of the specification, so a port listed in an unusual format is missed and a document can produce a false finding. Neither check costs tokens itself; M's extra cost is the extra rounds of reflect and edit it causes.

Beside it: Claude Code as the whole agent

Here the whole agent is swapped. Claude Code runs headless (claude -p) inside the harness's container, with file and shell tools and a 60-turn budget, on four Claude models, one run each, on the same 86 problems and the same 480 s wall cap. Each is paired problem by problem with M3c's arm A (LangGraph with gpt-6-luna, 3 runs, 61.6%). It was not pre-registered.

• Not in order of model size. Opus 73.3%, Sonnet 69.8%, Fable 67.4%, Haiku 66.3%.
• Fable is held back by the wall cap. It stopped on the 480 s cap on 17 of 86 problems, and 10 of those failed; its median wall time is 265 s against 24 to 83 s for the others. Under this cap its score is a lower bound, not a measure.
• Two things change at once. Each arm differs from the base in both the harness and the model. The LangGraph agent with a Claude model was not run.
• Cost is not comparable. These runs used a subscription plan, so no call has a provider-reported price. Claude Code's own estimates at API rates are in the report, labelled as estimates; they are not money spent, and are not drawn here.
• At least one Claude model solved 74 of the 86 problems, and all four solved 45.

The problems no Claude model solved

M3d: verify before submit

Our agent submits after one turn on about half its problem-runs (47% on arm A) and runs no testbench of its own on 55%, because its reflect step accepts a clean compile, lint and synthesis. Claude Code iterates: it writes or runs a testbench and fixes what it shows. Does making our agent verify before it submits close that gap? M3d adds two fixed graph steps the model cannot skip, and runs each against a fresh base A, 3 runs an arm on the same 86 problems and settings as M3c.

• V, verify before submit. A submit is refused until the design, as it stands after the last edit, has passed a testbench the agent ran: its own, or one the task gives. After 2 refusals, or when no edit can follow, the submit goes through.
• S, V plus specification cases. Before planning, one LLM call writes cycle-level test cases from the task and its documents only. It never sees the RTL, the given testbenches or anything the agent wrote; a unit test checks that on the recorded request. Verify applies the cases too.

The failure mix moves, again

The step did what it was built to do. Failures with no testbench of the agent's own fell from 55 on A to 0 on V and 2 on S. They did not become passes. Failures where the agent's own testbench passed and the hidden tests still failed rose from 16 to 48 and 44, and cap stops from 7 to 27 and 43, as the extra turns ran into the 480 s wall cap. This is M3c's lesson a second time: a testbench the model writes agrees with the model's own misreading. The blind specification cases, written by the same model, carry the same misreadings.

• The verify step refused a submit on 155 of V's 258 problem-runs, mostly because no testbench had run on the current files, and the agent then edited. The refusals fell mostly on problems the agent was going to get right anyway.
• S's guard caveat. 8 of S's problem-runs were stopped by the whole-run cost guard, which charges the per-problem budget for every call whose cost was not yet reported, and S makes more calls. They count as failures, as pre-registered. Counted at A's pass rate instead, they would move S − A by about +1.9 points: still negative, and the verdict does not change.
• Without the 7 benchmark-limited problems (79 problems, beside the full-86 result, not instead of it): A 66.2%, V 64.1%, S 61.2%; V − A −2.1 [−7.2, +2.5], S − A −5.1 [−12.7, +1.7].
• The problems Claude Code won stay lost. On the 7 problems that 2 or more Claude models solved and M3c's arm A never did, V and S pass 1 and 1 problem-runs of 21, as A does (1): making the agent run a testbench is not what Claude Code did there. It wrote a testbench that checks the right behaviour.

What it means

On this model, changes to the agent's process are now null three times: a memory of lessons (M3b), a mutation gate and an interface check (M3c), and verify before submit (M3d). What has moved the score is the harness (Deep Agents, M3b, confirmed by a pre-registered run) and the model (the Claude Code ladder, where harness and model change together). That is one model, gpt-6-luna, on one benchmark, at a resolution of a few points. It does not say that process never matters; it says that, here, checks the model writes for itself do not catch the model's own misreadings.

Why it fails

Every failed problem-run in M3's 14 runs gets one label: the first of a fixed list of rules that holds, read from the harness's per-test log, the agent's trace and its summary. The rules are in tools/taxonomy.py; no model is involved.

Counts are recomputed from the failure_label column of the committed CSVs and match the M3 report.

The interface story: a handle the agent was never given

Two labels are interface failures: the hidden build failed on code that compiled for the agent (74 failures, on 8 problems), or the hidden testbench reached for a port, signal, instance or parameter by a name the design does not have (73 failures, on 9 problems). The agent cannot see either from its own checks, because the names are only in the hidden testbench.

trace: compile exit=0 iverilog -g2012 -o /code/rundir/agent_tb/compile.vvp -I/code/rtl /code/rtl/csr_a
trace: lint exit=0 verilator --lint-only -Wall -Wno-fatal -Wno-MULTITOP --timing -I/code/rtl /code/
trace: synth exit=0 yosys -q -p read_verilog -sv /code/rtl/csr_apb_interface.sv; hierarchy -auto-top
trace: llm reflect decision="submit"
log:   AttributeError: csr_apb_interface contains no child object named pclk. Did you mean: 'clk'?
log:   ======================== 2 failed, 3 warnings in 1.13s =========================

For 6 of the 9 missing-name problems, the main name the hidden testbench reads is not in the prompt or in any file the agent is given. No change to the agent can be expected to match it. Those failures belong to the benchmark items, and M3b leaves them out of its evaluation set.

*A common word that may not name the signal. "Errors" counts every AttributeError line over the 14 runs, so a parametrised test counts many times. The search is a word search of each problem's prompt and context files; it is not yet a committed tool.

Self-check passed, hidden tests failed: 61 of 73

When a submission failed and the agent had run its own testbench on it, the last run of that testbench had passed in 61 of the 73 cases. Its testbenches check less than the hidden ones. These 61 failures are spread over 26 problems.

Read as a verification engineer would

The hidden cocotb tests play the role of a sign-off regression the designer cannot see. The agent's own testbench plays the designer's unit test. Mapped onto the usual verification concepts, here is what the project does and does not do.

The two M3 failure modes have verification answers. A testbench that passes on wrong code is a testbench-quality and coverage problem: a mutation score asks whether the testbench fails on mutated copies of the design, and so measures what it can catch. A name the hidden testbench needs but the design lacks is an interface problem: elaborating the design and diffing its ports, parameters and hierarchy against the specification catches it before submission, when the name is given. Neither has results yet.

How this compares with others

Every number here was checked against its primary source on 24 October 2026. Our runs use only the free, open-tool split (92 problems, four categories), specific models, one run per setting in M2 and pass@1 from a single sample.

M3b's four arms (57.8% to 65.1% pooled pass@1), its confirming run, M3c's three arms and the Claude Code ladder (66.3% to 73.3%) are not drawn beside these rows: they were scored on 86 problems, the 92 minus 6 that no agent can reach, so their rates are higher by construction and do not sit on the same 92 as the LBC-bench rows. The LBC-bench row for Claude Code (Sonnet 4.6) is a different model and a different run from our ladder.

One further paper evaluates small models with an agentic framework on CVDP (Shankar et al., arXiv:2512.05073); its abstract gives no pass rates, and none are quoted here.

Threats to validity: what this does not show

• One model per arm. Every M3, M3b, M3c and M3d arm uses gpt-6-luna. An ablation that does nothing for it may matter for another model.
• Sampling is not seeded. CVDP_SEED is a label only; OpenRouter sampling cannot be seeded across providers. Both runs of every M3 arm carry the label seed 1.
• The free, open-tool split only. 92 problems in four categories. The verification-writing categories (testbench stimulus, checkers, assertions: cid012–014) need Cadence Xcelium and are not run.
• 2 runs per arm resolve about ±5 points in M3, and M3b's 3 runs on 86 problems do little better. A real effect smaller than that would read as "no detectable effect". The null results are not evidence of no effect.
• M3b makes four comparisons with three runs an arm. None of its sign tests passes a Bonferroni threshold of 0.0125, and the bootstrap intervals are not adjusted. The confirming run re-tested only B − A, as one pre-registered test (+4.7 [+1.0, +8.5], confirmed); D − C was not re-tested and stays a hypothesis.
• The confirmation is one model and one benchmark. gpt-6-luna on the 86 free agentic problems, with one Deep Agents version. Its sign test (p = 0.093) is moderate evidence; the rule read the interval only.
• Every Claude Code ladder arm is one run, and the ladder was not pre-registered. Read its effects against the base's run-to-run spread, on top of the interval shown.
• The ladder changes harness and model together. Claude Code with a Claude model against LangGraph with gpt-6-luna, with a 60-turn budget against 10; it does not say which change is responsible. Its plan has no per-call price, so it has no comparable cost.
• M3c's gate fails open when it cannot mutate. 61 of 390 gate runs were skipped and let the submit through, so M is not a pure "every submit was gated" arm.
• Some problems are benchmark-limited. 7 of the 86 evaluation problems need a detail the task never gives (a file name, default parameters, a lookup table), shown by counterfactual reruns. No agent can be expected to pass them, so they cap every score and shrink every effect a little. They are kept in the reported scores, and the 79-problem figures are shown beside them, never in their place. reachable.py, which built the 86, checked the names the hidden test reads but not the file paths it compiles.
• M3d's specification cases are written by the same model. The spec call is blind to the design, not to the model's way of reading a specification, so its cases share the agent's misreadings; a different, stronger model for that call was not tested. M3d also kept M3c's 480 s wall cap, and much of V's and S's loss is cap stops: a larger cap would be a different experiment.
• M3d's S lost 8 problem-runs to the whole-run cost guard, which charges unreported in-flight calls. Counted at A's pass rate they would move S − A by about +1.9 points, not enough to change its verdict.
• M3b tests one memory snapshot. 40 generic lessons from 87 attempts, only 19 of them passes, several near-duplicates, 6 never retrieved. A null C − A says this memory does not help, not that no memory could.
• M3b's caching claim is inferred. Cached prompt tokens are not recorded per call, so the reading that Deep Agents' extra tokens are mostly cached rests on its cost per prompt token, not a measured cache-hit count. Its rounds are also not turns: compare the harnesses on cost and wall time, not on turns.
• The M3b effects are paired on the 86 reachable problems. On all 92 the 6 excluded problems fail in all 72 of their runs, so every effect is a little smaller and no finding changes (B − A +4.0, C − A +1.4, D − C +5.4 points).
• The hand-label hold-out had no signal_missing_hidden case, so the new rule is checked only on the set it was developed on, and the same person wrote the rules and the labels.
• Problems are treated as exchangeable by the bootstrap and the sign test. cid003 (34 problems) weighs most.
• Self-check pass or fail is read from truncated tool output in the trace.
• The hidden build's own error text is not in the log, so a likely cause (a renamed or missing parameter on phase_rotation_0028) is not shown for each problem.
• "Is the name given to the agent?" is a word search over prompt and context files. It cannot tell a signal from the same word in prose, and it is not yet a committed tool or done for compile_fail_hidden.
• No per-category or AMBA effects. On the 8 AMBA problems every run passed 3 or 4; two runs give no power on 8 problems.
• One run per runner in M2. The seeds measure the noise on 30 problems, not on 92, and the reasoning setting for each model was picked from one run on 10 problems.
• The heavy split (75 problems) is never scored with the agent. In 46 of them the files the reference solution changes lie outside the agent's edit roots (M1 gave 53 from a check it did not commit; M3b's committed tools/learn_in_roots.py gives 46). M3b runs the agent on the other 29 only to build its lessons memory, and 3 of those (i2c 7, 46, 49) gave no test output in any attempt.
• Some costs are lower bounds: one unreported call in M2 (DeepSeek seeds), one in M3 (arm C), 2 in the M3b confirming run, 4 in M3c and 37 in M3d.
• Caps are soft by design: wall time by up to about 40 s, the budget by up to one call.
• Model serving varies: DeepSeek's calls went to 15–19 providers per sweep, so its cost and behaviour mix providers.
• One log label is wrong but harmless: tools/log_check.py called a simulation that hung "harness_error" (4 in the first two M2 sweeps); the pass rates are right.

What's next: open questions

Questions a reviewer might ask

Glossary