69 Tests. All Passing. Zero Bugs Caught.

The useful part

Together they caught zero of the eleven bugs I had deliberately planted in that module. A second setup, pointed at the specific bugs rather than at the module, used 17 attempts and caught ten. It changes source code in small ways, flips a comparison, alters a constant, deletes a raise, then runs the existing test suite and records which changes the suite fails to notice.

How it works

• A toy module with one happy-path test showed 47% line coverage and caught 2 of 21 mutations.
• This tells you whether anything would have failed if that line had been wrong.
• For each one it writes a single test, and that test is kept only if it both passes on clean code and fails on that specific mutation.
• Three approaches, same model, same token ceiling, twelve widely used Python libraries including cachetools, toolz, tenacity and boltons.
• 455 mutations generated, 53 of which survive the existing tests and sit on lines those tests actually execute.

What to take from it

approach caught one prompt, "write more tests" (556 tests) 9 of 53 one test per call, no targeting (53 tests) 2 of 53 targeted at the specific fault, with a pass/fail gate 44 of 53 I hand-audited the nine it missed. Seven are provably unkillable, six of those being type annotations inside TYPE_CHECKING bloc...

Example or evidence

Details worth keeping

A change nothing catches is a fault your tests cannot detect. Then I pointed an agent at the mutations the tests missed.