Contamination-Resistant Code Evaluation
Static coding benchmarks degrade as their tasks leak into training corpora — scores partly reflect training-set memory, not capability. W2 sidesteps that failure mode by synthesising evaluation tasks directly from real, versioned repositories via AST analysis: tasks become a pure function of (commit, extraction rules), so contamination turns into a property that can be reasoned about and refreshed, rather than an invisible confound.
Working paperpilot measured — operational results only · resistance study next
The problem
Benchmark validity is tied to how far the test data has escaped the training set — and for public coding suites, the honest answer is: not far.
Early execution-based suites established functional-correctness testing but are single-file, synthetic, and now heavily leaked into training corpora; hardened variants added mutation-generated tests without solving contamination. The field’s response has taken three forms: temporal harvesting (collecting problems published after a model’s cutoff), private held-out repositories, and automated generation from real repositories. W2 belongs to the third family — distinguished by two deliberate constraints: it runs end-to-end on a single 32 GB GPU, air-gapped, and it targets a security-relevant validation library, where a hallucinated “fix” is maximally costly.
Two lessons from that literature shape everything below. First, contamination resistance is only established by a fresh-versus-stale comparison — not by the mere act of generating from a repository. Second, the field’s standard for a code “pass” is execution-grounded: success means tests pass after the model’s change, not that the output reads as plausible.
Method — tasks from a pinned commit
A structural extraction engine reads a target repository and manufactures fresh tasks bound to that code’s actual signatures, imports, and edge cases — up to one task per category per file.
Tasks are a pure function of (repo state, extraction rules) — re-derivable from the exact commit months later, by an independent party. That is the substrate contamination control is built on.
The pilot, read honestly
Sixty tasks — twenty per category — ran end-to-end against Cerberus, a security-relevant input-validation library. Every task completed. And that is precisely why the result is not a capability score.
By the discrimination rule this lab applies to every rubric — a task set on which every sample passes measures nothing — a 60/60 result is not a capability claim. It says the generated tasks sat below the model’s ceiling. That is the pilot’s central finding: the extraction pipeline runs end-to-end, deterministically, within the VRAM budget — and the difficulty distribution must be raised before any number carries signal. What is trustworthy here is the operational measurement: throughput, stability, token accounting under a pinned single stream.
What the pilot does not show
Two properties the run does not establish — named precisely, because each becomes a measurable criterion in §05.
Five falsifiable criteria
The workstream hypothesis: tasks regenerated from live repositories resist memorisation — a model’s advantage on seen code does not transfer to structurally-matched unseen code drawn by the same pipeline. Every claim carries an instrument and a threshold, marked predicted until measured.
C1 and C2 convert the pilot’s honest boundaries into measured results; C3–C5 formalise the next-steps list. All five run on the same single-GPU, air-gapped setup plus the W1 pipeline already in the program — which is the point: the whole plan is verifiable under the sovereignty constraint the pilot was built to honour.
Applied scenario
Vetting an open-weight model for an air-gapped, security-critical codebase — the use case the method exists for.
- Situation. A team maintains a proprietary input-validation library under a data-sovereignty constraint — nothing may leave the network. Public coding benchmarks are contaminated, so a leaderboard rank is no evidence for this codebase.
- Task. A defensible, on-premises measurement of whether a candidate model handles this repository’s edge cases and injection-vector reasoning — reproducible from the exact commit, inside one 32 GB GPU, with no cloud dependency in the evaluation path.
- Action. Point the AST extractor at the pinned commit; synthesise tasks bound to the repo’s real signatures and imports; run deterministically at temperature 0 under a pinned single stream; log latency, tokens, and finish reasons per task.
- Result. A complete evaluation in under five minutes, entirely on-premises and re-derivable from (commit, extraction rules) — defensible in an audit months later. The deliverable is a trustworthy, sovereign measurement and a precise instruction for the next iteration — not a number taken on faith.
Limitations, stated up front
- A different task family. AST-synthesised tasks are structurally cleaner than issue-resolution tasks (“resolve this GitHub issue so its hidden tests pass”) — the pipeline complements rather than replaces issue-based evaluation, and its difficulty ceiling is one reason the pilot saturated.
- Single operator, single hardware. The operational numbers (throughput, latency) carry that external-validity bound independent of the capability question — a C4 concern by design.
- Identification is not exploitation. The security_audit category currently scores identification; a finding that reads as correct may not be — which is precisely why C2’s reproduction requirement is load-bearing for that category.
- Quantisation is the open caveat. Q4_K_M preserves surface validity more reliably than deep multi-step reasoning — correctness under low-bit weights is what execution grounding must confirm, not assume.