Field notes
Thinking out loud from the corral
Notes from the corral: what auditing tests by execution has taught us, the runs behind the claims, and what goes on the record.
One workflow file
The same Warp crate, audited again, this time by corral's GitHub Action on GitHub's own runners from a fork: a released version, one workflow file, and a key in a repository secret. The crate's tests caught 7 of 37 planted faults; for 27 of the 30 that got through, corral wrote a test that catches it and ran it. Getting there took a release and four fixes, every one found by running corral the way a stranger would.
Read the note →The first Rust audit
corral can now audit Rust. Its first run on someone else's code was one file of Warp's fuzzy matcher: the crate's own 33 tests caught 11 of 34 planted faults, and for 22 of the 23 that got through, corral wrote a test that catches it and ran it to prove so. Here is how a test lands in a Rust project, what the run measured, how we checked it, and what it does not show.
Read the note →One question per test
corral's test-critic used to hold a conversation: read the tests, file findings one tool call at a time, stop when done. In v1.0.0-rc.16 it answers one typed question per test instead: a framing TypeSafe's Jev made loud, on a pattern the local-model community has used for a long time. On vacuous tests planted in seven of our own test files it caught 63 of 63 to the loop's 36, on about a quarter of the input. Here is the decision model, where it came from, how we measured it, and what the measurement does not show.
Read the note →The price of a complete answer
We made corral's test-critic hand its answer to a schema, and then made it judge every test by name. On a cloud model both changes were free. On a local model the first one let it quietly skip most of the tests, and the fix for that cost it accuracy. We shipped the fix anyway. Here is the measurement and the reasoning.
Read the note →For every season, churn
Corral's review loop finds real defects, and the fixes for them keep producing new ones. We measured the ratio, changed how we fix, and then watched it happen again this week, in a merge-gate fix and in a release-gate fix that had sat untested for 25 days.
Read the note →A ledger, not a log
Corral stores every verdict as a signed, hash-linked file on an orphan git branch, and any DuckDB reads it in place. The pattern is not new — an existing GitHub Action has kept benchmark history on a branch for years. This is what the chain buys over that, what it costs, and the honest state of extracting it.
Read the note →The fixer is the worst witness
Seven rounds of cold review on corral in one day. The findings were real, but the pattern underneath them is the useful part: a rule enforced at one door and not the others — and round after round, the thing it kept catching was my own fix for the previous round.
Read the note →Two videos, and what they do not show
Sixty seconds of an audit and eighty of a review, both real runs on a library we did not write. This is the annotated version: what is happening in each frame, what the pipeline applied, which numbers you can check against the public record, and the three claims in the review that our own harness threw out.
Read the note →The auditor never builds
On the day the seventh release candidate was cut, the builder came back as a question: Claude builds, hands the pull request to the adversarial auditor, the reviewer reviews, the audit goes back to Claude. Most of that loop already exists. We said no anyway, and the reason is the thesis: an auditor that also builds has the exact conflict corral exists to remove. A field note on the design shift of the last four days — the review loop went from a design to the front door — and on the one word that stays out of the product.
Read the note →The ledger is just text
We stopped storing verdicts in a database. A corral run now writes one signed, hash-linked JSON entry into a directory — a git branch on GitHub, a folder on a laptop — and DuckDB reads the directory as a table. Twenty-one kilobytes a run, a chain you can verify with sha256, and a query engine that never owns the record. A field note on what that changes, what it borrows, and what it is not.
Read the note →Grade the reviewer
Corral grades a model's tests by execution. This week's cold reviews made something obvious that should have been obvious a year ago: a model's reviews can be graded the same way — by whether its claims reproduce, against other reviewers, and against the signed audit of the very code it reviewed. A design note, written the morning after the aha.
Read the note →We put five strangers on our own code
Eleven pull requests in three days, every one of them a defect found by a model that had never seen the repository — including a 'proven' bug that could be fabricated and a trust anchor that shipped with a default. A field note on cross-model adversarial review: what it found, what it got wrong, and the one time we overruled a reviewer who was right.
Read the note →What is actually new here, and what is not
Mutation testing is fifty years old and its cost problem was solved twice. A four-hour run on psf/requests made us ask whether corral is breaking new ground or re-deriving it. The honest answer is two firsts, one stance, and a lot of known ground reassembled with its receipts — and a reason the seat was empty.
Read the note →Half a million tokens to spot a one-line change
The first time our cost meter ran, it printed one line about our own tool that nobody had ever been able to see: the test-writing model was being handed the same 36 KB file twenty-four times per call. What happened in the twelve hours after that line, and the rule it left behind: cold exam, warm prover.
Read the note →The database found the bugs
We made our audit tool write every decision — every planted bug's fate, every model call's cost, every downgrade's reason — into a DuckDB file. Within 48 hours the database had caught five bugs in the tool itself that no test suite had. A field note on analytics as a debugger, and why the receipts now live in a warehouse you own.
Read the note →Local models break things well and test them badly
We ran the same 9B model in two seats of the same audit. As the bug-planter it beat a human test suite thirteen times. As the test-writer it could not produce a file that compiled. That asymmetry decides where a local model belongs — and it happens to put the free one in the expensive seat.
Read the note →The gate is whatever you happened to install
Three times this week we found corral crediting a test suite with catching a bug it never detected. Fixing the third one exposed the shape underneath: the check that decides whether a mutant counts at all depends on what is installed on the machine — so the same audit can report a different denominator on two boxes.
Read the note →Eighty-eight tests that were not there
We pointed corral at three third-party Go libraries with good reputations and real test suites. It planted bugs their tests missed, then wrote and RAN tests that catch them — 88 of those, across 15 files. Here is the whole result, including the third of the panel it could not grade at all.
Read the note →Both models missed the same nine
Corral's design document has claimed a decorrelated herd for months. Nothing measured it. We built the measurement, pointed it at two writers from different labs, and got a Jaccard of 0.750 over their misses — of everything either one missed, both missed three quarters of it.
Read the note →It could break the code but not test it
The same local model, in the same run, planted 13 bugs that defeated a real test suite and then could not author a single compiling test. Breaking and checking are not one skill, and the seat you put a small model in decides whether it is useful.
Read the note →The compiler was doing the killing
Corral signed a record saying a test suite caught 10 of 13 injected bugs in a file its tests never execute. The suite caught none of them. A mutant that fails to build makes the test command exit non-zero, and that was being scored as a kill — so the number the whole product rests on was crediting the compiler's work to the tests.
Read the note →Four green checks that measured nothing
A week of pointing corral at corral. The security gate that never ran its scanner, the fault highlight showing unmodified code, the critic that reported a clean review while switched off, and the audit gate blind to the one change it exists to catch. Every one looked like a working check from the outside.
Read the note →Prove your tests would catch a real bug, by breaking your code
Your suite passes. That tells you the tests ran, not that they would notice if the code were wrong. Corral answers the question by execution — and hands you the test you were missing.
Read the note →The audit question nobody can answer yet
AI-authored code is entering regulated systems, and no one can show that the model which wrote it did not also grade it. Corral enforces that separation and attests to it — here is exactly what the artifact proves, and what it does not.
Read the note →The fourth participant
corral keeps three models apart so none of them marks its own exam. It never asked who wrote the code — and if that was the same model, the judge is grading its own work. This was the default configuration.
Read the note →Fifty-one minutes
I was going to launch this morning. With the window still open, I finally ran corral against one of my own projects for the first time — and found out it couldn't audit it at all. The post I had written said it supported five languages.
Read the note →The knife was ours
We told Sakana they shipped the capability and left out the containment. Today I found the same shape in my own code: the router that makes 'no model marks its own exam' true could only ever move one seat of three. Named, fixed, and proven cross-vendor.
Read the note →Four words that cost me a day
corral finally proved real bugs in someone else's code — reliably, ten times out of ten. Then it spent an afternoon telling me an audit had failed without telling me why. Plus an honest accounting of how far 'audit a whole repository' actually is: on a 236-file project, corral can currently see nine.
Read the note →A judge may not certify herself. So I made corral certify corral.
The whole thesis is that no one can judge their own work. So I turned the tool on itself and asked it to audit its own kill-rate scorer. It couldn't — for six reasons, each hiding behind the last, all wearing the same mask. Here's the debugging, the six fixes, and the moment it wrote a test for its own code.
Read the note →You can delegate the labor. You can't delegate the accountability.
AI will write your tests now — by the thousand, all of them green. That feels like relief. It should feel like a question: if nobody read them and nobody cares what they do, what is actually deciding your software is fit to ship? A short answer, and a command you can run against your own code in a few minutes.
Read the note →The code is the code
We pointed corral at a real, respected, well-tested project. Its suite killed all eight planted bugs — and then our own critic flagged four of its tests as worthless. The tempting move was to hide the criticism and publish the flattering number. We didn't, and chasing down why taught us the one rule a tool built on trust cannot break: it may only assert what execution proves — including, especially, when the thing making the claim is itself an AI.
Read the note →The critic was never the point
We opened by telling you your tests suck, then spent a note learning to keep that voice honest. Both were about the critic. But the critic was only the door. The point is friendlier and, in the agentic era, urgent: an AI will write you a thousand green tests in an afternoon, and the number of passing tests has never been what makes software stronger — the number that verifiably test something is. corral makes your suite stronger from both ends: it hands you the killing test you were missing, and it proves, by execution, which of your thousand green tests have decayed into no-ops you can delete. And it's built so everyone's tests get stronger from a shared corpus of findings that are verified and signed — knowledge you're actually allowed to trust.
Read the note →Good baking means always mind the D'oh
A builder is bounded by what a model can write. A certifier is bounded by what its sandbox can actually run. Those are different constraints — and confusing them meant we spent a while certifying Go over and over while telling a five-language story. Here's the mistake, the sharper mistake hiding under it, and why the whole thing tastes better for the kneading.
Read the note →We ran our own gate as a stranger (and it bit us)
Nemo iudex in causa sua — no one may judge their own cause — is the whole thesis of the tool. So we pointed corral certify --local at a Python file as a first-time user, not as its authors. It caught the flaw it was built to catch. It also failed its own onboarding in a way only a stranger would ever hit: the absolute path our own README tells you to type. Here's the real run, the Frampton Flaw it found, and the two bugs we found in ourselves fixing it.
Read the note →Nemo iudex in causa sua
No one may be judge in their own cause. It's a maxim of natural justice — and it's the whole design. Most ways of checking AI-written code quietly violate it: the thing that did the work reports whether the work is good. Corral is built the other way, three enforced constraints deep — and here's the objection a good CISO raises, and the answer.
Read the note →Nobody Fails a Test They Never Took
We borrowed one idea from Fugu: route each task to the model that's measured-best at it. Then we built the measuring, and learned that measuring fairly is the entire problem. Three times we penalized a model for the wrong thing — a task it never ran, a success that wasn't its doing, a verdict a human hadn't blessed — and what a fair fitness signal actually takes.
Read the note →Terribly sorry to trouble you… but your tests, um, well. They suck, really
Roger Ebert collected his most withering pans into a book called Your Movie Sucks; John Cleese built a career on the flaw everyone can see and no one can bring themselves to name. I built that critic for your test suite — it mutates your code, runs your own tests against the bugs, and if the tests don't scream it says so (terribly sorry to trouble you), with a signed record and a decorrelated second opinion. Three real verdicts follow: a two-star, a needs-review, and a genuine zero-star.
Read the note →Killing the part I was proudest of
A pivot you don't commit to is just a mood. Three days ago I wrote that corral was an accountability engine, not a builder. Then a friend reviewed the code and emailed me the bottom line: an interesting mashup of the LLM 'harness' and 'router' worlds, with a couple of fun/novel features. He was right — and the couple of novel features were the only part with no incumbent. So we're deleting the builder. Here's why the demo everyone loved had to go, and what corral actually is now.
Read the note →Is any of this worth it?
I lay awake asking whether a very cool thing was a useful thing. By morning the answer was no — not the way we'd been aiming it. We can't win as a builder. But the engine we built to contain agents and run their checks isn't a builder's engine at all. It's an accountability engine. Here's the pivot, and the working proof it produced.
Read the note →What we took from Fugu — and why
We dinged Fugu for shipping the capability without the containment. Fair's fair: here's what it got right, the idea we took from it, and the one honest line about how we took it — the insight, not the mechanism.
Read the note →Fugu, and the knife it leaves unwashed
Sakana's Fugu makes the case that orchestrating frontier models is its own scaling axis. It also leaves out the one thing its own name is about — containing the poison. Field notes on where we agree, and where we diverge.
Read the note →We caught our own gate cheating
The whole pitch here is a deterministic gate you don't have to trust. Then a pre-launch audit found the gate trusting the one thing it never should: the worker's word. Here's the bug, the failing test, and the fix — while nobody's watching.
Read the note →Why we built a brain, not a bot
Agentic coding got real in 2026 — and stayed a black box. We wanted the opposite: a herd you can contain, certify, and query. This is the founding why, and why the whole thing sits on one queryable ledger.
Read the note →