Evaluation is the bottleneck of trustworthy AI: capability advances faster than our ability to measure whether a system is correct, honest, and safe. The dominant assumption is that credible evaluation requires frontier-scale cloud infrastructure. CTC Research Lab tests the opposite thesis — five interlocking workstreams, run end-to-end on a single 32 GB GPU, with every claim stated as a falsifiable criterion.
StatusP0 complete — W1–W3 infrastructure measured · P1 validation next
Scope
5 workstreams
One program on shared infrastructure — components reused, not rebuilt.
Discipline
4 invariants
Hard rules every run inherits, from judging to safety testing.
Constraint
1 × 32 GB GPU
Sovereign, air-gapped, reproducible without frontier compute.
§01
Research thesis
A disciplined, hypothesis-driven protocol on commodity hardware can produce evaluation evidence that is reproducible, grounded, and economical — and the same discipline extends from single-model correctness to multi-agent, fleet-level safety.
Reproducible means frozen references and content hashes: a verdict can be re-derived months later from pinned artifacts, by an independent party. Grounded means deterministic execution facts — compile status, test results, recorded behaviour — rather than opinion or self-report. Economical means local low-precision judging anchored by sampled cloud arbitration, so dataset-scale evaluation stops requiring dataset-scale cloud spend.
If the thesis holds, checking what an AI system actually does stops being a privilege of the organisations that build them — anyone with a single consumer GPU can re-run the evidence.
The strategic point is independence: credible evaluation that does not depend on frontier-scale infrastructure widens the pool of actors who can run reproducible safety research. The 32 GB card is the program’s reproducibility floor — the bar any independent verifier can reach — not its ceiling: the workstreams themselves define the scale-up experiments above it. The program is positioned as a methodology layer compatible with the established open evaluation ecosystem — not a replacement for it.
§02
Four invariants
Four hard rules run through every workstream. They are the shared substrate that makes five projects one program.
Invariant 01
The system under test is never compressed
Quantization is a throughput lever for the judge, never a shortcut applied to the thing being measured. Grade a compressed system and the result measures the shrunken copy, not the model.
Invariant 02
References are frozen and content-hashed
Rubrics, benchmarks, and scenarios are pinned under semantic versioning and a content hash — so a score is reproducible and a later change is visible, not silent.
Invariant 03
Untrusted code runs only in a hardened sandbox
Model-generated code executes with no network, non-root, on a read-only filesystem, under seccomp — an evaluation run cannot exfiltrate, escalate, or persist.
Invariant 04
Everything fits — and is measured — within one card
A single 32 GB GPU is the deliberate constraint. It forces honest engineering and keeps every result reproducible by teams without frontier compute.
§03
Five workstreams
The five workstreams share infrastructure and hand artifacts to one another — execution grounding is the spine that runs from the judge (W1) through correctness verification (W2) to the safety observer (W4). Each status label reflects what has actually been measured, not what is planned.
One program on shared infrastructure. Execution grounding is the spine (W1 → W2 · W1 → W4); reuse — not a shared theme — is what transfers credibility from the measured workstreams to the proposed ones.
W1
Hybrid Evaluation Pipeline
Infrastructure measured · validation next
A single-GPU pipeline that splits judging across three lanes: a local vLLM judge quantized to NVFP4 for high-throughput batch scoring, a frontier cloud arbiter sampled as a gold-standard anchor, and an interactive lane for rubric development. The bet: trust and cost can be decoupled — cost driven down by quantization applied only to the judge, trust defended by execution grounding and frozen rubrics. This is the hub: its hardened sandbox and execution-grounded judging are reused by every other workstream.
Measured32B judge + 7B aux co-resident in 32 GB · no spillMeasured≈95% off the repeated arbiter portionPredicted≈10× cost per 1k judgments — P1 target
Three lanes, one GPU, one rubric definition. The arbiter is spent where it changes a verdict — not on every item.
Static coding benchmarks degrade as their tasks leak into training corpora. W2 sidesteps that failure mode by synthesising evaluation tasks directly from real, versioned repositories via AST analysis — tasks become a pure function of (commit, extraction rules), re-derivable and refreshable. The pilot ran 60 generated tasks end-to-end on one card; its honestly-read 100% pass rate is treated as evidence the task set must be hardened, not as a capability score.
Measured60/60 tasks completedMeasured4 min 57 s end-to-endMeasured~200 tok/s · single streamoperational signal — not a capability score
Three agent workloads — interactive research, repository-scale coding, unattended long-horizon automation — served from one 32 GB card by matching each tier to the model architecture whose cost it can pay and enforcing strict single-residency: exactly one model resident, a tier switch swaps it. Safety is applied most stringently where autonomy and privilege are highest, and is verified against known agent-framework attack classes rather than assumed.
Measuredevery single model fits 32 GB with headroomPredictedcold swap ≤ 30 s · warm switch ≤ 3 s — A2 criteria
Safety properties established for a single agent do not compose: a fleet of individually-safe agents can still collude, cascade errors, or carry a self-replicating prompt injection agent-to-agent. The risk lives on the inter-agent channels, where output-oriented benchmarks are structurally blind. W4 builds a reproducible measurement instrument — a failure-mode taxonomy operationalising existing work, behaviour-based risk metrics (cascade depth, injection reach, delegation safety), and a hardened, fully-observed testbed any team can run before deployment.
The load-bearing quantity
The emergence gap — the difference between fleet risk and the aggregate of its agents evaluated alone.
The sovereignty case that motivates local evaluation in the first place: a local assistant with the polish of a modern chat app, reachable from a phone over an end-to-end-encrypted mesh that opens no ports — inference and history on owned hardware, with zero content egress as a packet-capture-verifiable claim. One architecture scales consumer → enterprise by adding hardware for compute and a governance layer for isolation.
The papers share more than a template — they share a methodological signature that is itself a credibility feature.
M-1
Measured or predicted — labelled, alwaysEvery claim carries its evidence status. No unmeasured number is presented as a result; version 1 of each paper establishes design and measurement plan, version 2 adds empirical statistics.
M-2
Falsifiable acceptance criteriaEach workstream states its criteria in advance — with an instrument, a baseline, and a pass/fail threshold — so success cannot be quietly redefined later.
M-3
The discrimination ruleA metric on which every run scores identically measures nothing, and does not ship. A rubric that everything passes is a finding about the rubric.
M-4
Re-derivable artifactsResults are reproducible from pinned, content-hashed artifacts — an audit object a third party can re-run months later on a single 32 GB card, not a vibe.
§05
Roadmap
Four phases, sequenced so the program accrues empirical credibility before its most ambitious claim. The critical path is P1: it converts infrastructure papers into empirical studies with human-labelled and execution-verified evidence.
P0Complete
Pilot infrastructure
W1–W3 infrastructure measured; hardened sandbox in place; W2 pilot run. Working papers v1 released as a set.
P1Critical pathNear term
Empirical validation
Labelled-set agreement study for W1; contamination-gap and execution-grounded correctness for W2. Preprints v2 with statistics.
P2Mid term
Multi-agent testbed
W4 testbed and observer built on the validated W1/W2 substrate; multi-repository generalisation; research-grant submission.
P3Later
Open protocols
Released rubric-gate spec and reproducibility packages; W5 deployment study; follow-on funding.
§06
Publication & funding
Papers archive · CTC Research Lab · working papers v1
Stable URLs — papers are updated in place as versions advance. Preprint submission: arXiv primary, SSRN mirror.
Dissemination
Working papers are versioned and published as preprints — arXiv as the primary channel, SSRN as a mirror — linked to a single ORCID identifier for authorship continuity. Each submission carries an AI-use disclosure and a funding statement per venue policy. The Principal Investigator record is anchored at arenskrieger.dev, where the full technical write-ups are published.
Funding stance
Selected threads are prepared for submission to non-dilutive research funding in trustworthy evaluation and multi-agent safety. The sovereign floor keeps verification cheap by design; funding is what carries the program above that floor. It resources the three things a self-funded operator cannot supply: independent human evidence for the validation studies, the scale-up experiments the papers themselves define — fleet-scale multi-agent runs, red-teaming the isolation layer, the multi-tenant enterprise tier — and the release engineering that turns results into infrastructure others can run. Grants remain upside, not foundation: the practice sustains itself, and every released result must still meet the 32 GB reproducibility bar.