CODE TO CAPITAL Research Lab · W1
Workstream W1 · CTC Research Lab

Hybrid Evaluation Pipeline

Evaluating frontier agentic-coding systems at dataset scale forces a trade-off between cost and trust: a frontier cloud judge is accurate but expensive per item; a local judge is cheap but must prove that quantization has not blunted its judgment. W1 resolves the trade-off by splitting work across three lanes on a single 32 GB GPU — and states every trust claim as a falsifiable hypothesis.

Working paperinfrastructure measured · judge-quality hypotheses in validation

Co-residency
32B judge + 7B aux in 32 GB
NVFP4 weights and a q8_0 KV cache hold both resident with ~7 GB headroom — no spill to system RAM.
Measured
Arbiter economics
≈95% off repeated portion
Batch API pricing plus a prefix-cached rubric preamble, reproduced on the target hardware.
Measured
End-to-end cost
≈10× per 1k judgments
Reduction target vs. an all-frontier-cloud baseline, at non-inferior agreement (H4).
Target · predicted
↓Working paper (PDF) →CTC Research Lab ↗PI write-up
§01

The problem

An evaluation judge is only useful if it is both cheap enough to run over large datasets and trustworthy enough that its verdicts carry weight.

The concern is not hypothetical: low-bit quantization is known to degrade precisely the procedural-reasoning capability an evaluation judge depends on — studies report degradations of up to ~32% on mathematical reasoning under aggressive post-training quantization, with the effect most pronounced on complex, multi-step reasoning. Judgment against a detailed rubric is a procedural-reasoning task. The fidelity of an NVFP4 judge is therefore treated here as a hypothesis to be tested (H1), not an assumption of the design.

Roughly 80% of the operational workload is batch evaluation of datasets against stable, frozen rubrics; the balance is designing two-page rubrics engineered to stress-test agentic coding systems. The contribution is not a new model but a measurable evaluation protocol — portable by design, implementable on top of established open evaluation frameworks rather than replacing them.

§02

Architecture — three lanes, one GPU

Bulk judging runs locally at FP4; a frontier arbiter is sampled only where a verdict is contested. One rubric definition governs all three lanes.

Frozen rubric · semver + content hash cached prefix — write once, reuse per item RTX 5090 · 32 GB · single GPU Local batch judge · vLLM · NVFP4 (~32B) ~80% of workload · dataset-scale scoring 7B auxiliary routing · co-resident Interactive lane rubric authoring · adversarial probing Hardened sandbox no network · non-root · read-only · seccomp execution facts — compile status · test pass/fail · linter — ground the verdict (H1) Cloud arbiter frontier batch API · cross-family sampled ~5–20% · anchor Precision rule System under test full weights — never compressed
Reproducible dataset-scale verdicts at local cost. The arbiter is spent where it changes a verdict — targeted at disagreements and borderline items, not uniformly.
The precision invariant

A model that is itself under evaluation runs at full weights — only a judge or an open-weight baseline may run compressed. Grade a compressed system and the result measures the shrunken copy, not the model.

§03

Five falsifiable hypotheses

Every trust claim is expressed as a hypothesis with a defined metric, an instrument, a baseline, and a target — so the pipeline's credibility rests on evidence rather than assertion. This paper deliberately reports no agreement statistics it has not measured.

H1 Execution grounding Supplying the judge with deterministic execution facts — compile status, test pass/fail, linter and type-checker output — increases agreement with human labels and reduces the false-accept rate on subtly-broken code. Doubly motivated: grounding may add signal, and it may be the compensation an NVFP4 judge needs for quantization-induced reasoning loss. Falsified if: grounded and ungrounded judging do not differ in Cohen's κ beyond the confidence interval. In validation
H2 Frozen-rubric reproducibility A rubric frozen under semantic versioning and a content hash yields high test–retest reliability across time and software updates — the mechanism that lets a re-run months later be compared to the first run at all. Falsified if: verdict agreement on re-runs of an unchanged rubric-plus-input falls below the pre-registered threshold (≥ 0.95 proposed). In validation
H3 Self-consistency as confidence Verdict variance across N stochastic judge samples is a calibrated proxy for reliability — high variance predicts lower correctness and triggers abstention or escalation to the arbiter ("trust or escalate", operationalised on a local judge). Falsified if: no monotone relationship between sample variance and error. In validation
H4 Tiered-cost efficiency Local FP4 plus a sampled cloud arbiter achieves roughly an order of magnitude lower cost per 1,000 judgments than an all-frontier-cloud baseline, at non-inferior agreement. The cost mechanism — batch floor and prefix-cache upside — is already reproduced on the target hardware. Falsified if: cost reduction misses the target, or agreement falls outside the pre-set margin. Mechanism measured
H5 No-spill co-residency A 32B judge (NVFP4) and a 7B auxiliary co-reside within 32 GB with a bounded KV cache and no spill to host RAM, sustaining batch throughput. Falsified if: observed spill or VRAM collision under the specified workload. Measured
§04

VRAM budget & cost cascade

Two numbers make the design work: everything fits in one card with headroom, and arbiter cost falls in stages rather than all at once.

VRAM co-residency budget · 32 GB · measured 32B judge · NVFP4 · ~16 GB 7B aux ~5 KV ~2 OS ~2 headroom ~7 GB q8_0 KV cache halves the 32B judge's cache from ~4 GB to ~2 GB at 16k context — the trick that buys the headroom. 0 ──────── 32 GB
Direct evidence for H5: peak VRAM below the ceiling, zero spill to system RAM under the batch workload.
Cost cascade · arbiter portion (relative) · mechanism measured 100% · interactive cloud baseline ~50% · batch API floor ~5% · prefix-cached rubric preamble per-item frontier calls batch pricing on the sampled arbiter rubric written to cache once, re-read per item
The local judge removes per-item cloud cost for the bulk; the arbiter portion then falls in stages. The end-to-end ≈10× claim (H4) remains a target until the P1 study.
§05

Rubric gates

Rubrics are not trusted until they survive a four-gate gauntlet — then frozen. A rubric on which everything passes measures nothing.

A frozen rubric plus a hash is the operational mechanism behind H2 — and the direct response to the benchmark-memorisation critique in the agentic-coding literature. Execution facts are supplied as outcome signals (compile status, pass/fail counts, linter classes), never as the reference patch, keeping the grounding gate and the contamination gate consistent.

§06

Roadmap — pilot to open protocol

P0 Complete

Pilot infrastructure

Measured co-residency and the cost model — the infrastructure results this working paper reports.

P2 Planned

Scale & generalisation

Multi-domain rubrics; arbiter-sampling ablation; external validity beyond one operator and one card.

P3 Planned

Open protocol & tooling

Released rubric-gate spec and a reproducibility package — seeds, hashes, container images, configs.

§07

Limitations, stated up front

Threats to validity — named, not buried
  • Correlated error. Judge and arbiter can share blind spots, so arbiter agreement is a calibration reference, not absolute ground truth. Mitigation: independent human labels anchor H1–H3, and the arbiter is chosen from a different model family than the judge.
  • Human labels are noisy. Annotators systematically over-reward confident, assertive outputs — label noise is quantified via inter-annotator agreement rather than assumed away.
  • Single operator, single hardware. External validity across GPUs and task distributions is untested (a P2 concern), mitigated by releasing a reproducibility package a third party can re-run.
  • NVFP4 fidelity is the open question. The assumption that a low-precision judge is "lossless enough" is exactly what H1 tests — the quantization literature gives concrete reason to expect a non-trivial effect.