What a test run can and cannot say about a patch

An agent hands back a patch and a green test run. What do you actually know?

You know that the patched code passes the tests. You do not know whether the tests would have failed without the patch. You do not know which parts of the patch the tests exercise. And you do not know whether some of the changes are simply along for the ride.

We expect this to matter more with agents than with people. An agent's patch can be large, it can include incidental edits, and the agent often writes the tests as well. Tests written at the same time as the code can pass for reasons that have nothing to do with the problem they were meant to catch.

Asking counterfactual questions

Instead of running the tests once, run them on variants of the code. Each variant answers a specific question. The proof protocol in DiffWitness asks five.

Contrast. Run the new tests against the old code. If they fail there and pass on the new code, they discriminate the problem the patch claims to fix. A test that passes on both says little about this change.

Necessity. Remove each production hunk in turn and rerun the evidence. If the evidence now fails, the hunk is witnessed. If it stays green, the hunk is unwitnessed. That does not mean the hunk is useless, only that the chosen evidence does not depend on it. If the experiment cannot support a conclusion, the hunk is inconclusive.

Sufficiency. Start from the old code plus the new tests, and add production hunks to find the smallest subset that passes. Small patches can be searched exhaustively. Large ones need a budget, and the result says which kind of search it was.

Interaction. Two hunks can back each other up. Remove either one and the tests still pass; remove both and they fail. Removing hunks one at a time would call both unwitnessed, so the pair is reported together.

Stability. Every variant is run more than once, twice in the default configuration. A flaky or timed-out result never supports a causal claim.

Reading a result

Here is the shape of what these experiments tell you, simplified for illustration. It is not DiffWitness’s actual output format.

contrast       old code: fail    new code: pass
auth.py  #1    witnessed
auth.py  #2    unwitnessed
util.py  #1    inconclusive (flaky)

The interesting line is the unwitnessed hunk. It might be a necessary change that the tests do not cover. It might be an incidental edit. It might be something that should not be there at all. The experiment cannot tell which, but it can tell you where the evidence stops. That is where a human reviewer should look first.

What it cannot say

None of this proves that the code is correct. Every result is relative to the evidence command you chose, so it cannot establish requirements that the tests do not encode. Bad tests, unreliable external services, hostile test commands and differences between environments can all mislead it.

The value is narrower and, we think, more honest. It makes it harder to pair a patch, by accident, with evidence that has nothing to do with it. And it points review attention at the parts of a change that nothing vouches for.

What we do not know yet

We do not yet know how much of a typical agent patch turns out to be unwitnessed on real repositories, or how to keep these experiments affordable on very large test suites. Both are on our research agenda.