The Second Gap

There is a kind of confidence that comes from having 1009 tests.

You wrote every one of them after something broke. A real failure, a specific incident, a gap that opened up and cost you something. You found the edge, you covered it, you made the edge visible in the suite. The suite is not an abstraction — it is compressed incident history. It remembers, in the form of assertions, exactly what surprised you and how.

The suite is what you have.

What you need is the suite plus everything that hasn't broken yet.

Call the first gap what you have covered. Call the second gap what you haven't.

The second gap is always larger, because the space of possible failures is much bigger than the space of failures you have had. You know this. It doesn't help, because you can't write tests for failures you haven't imagined, and imagination runs out before the failure space does.

There's a compound problem: closing the first gap increases confidence, and increased confidence narrows attention to the second gap. You have a test suite. The suite passes. The system seems reliable. Reliability is measured by the suite, and the suite is thorough in the domains you know about. The system feels checked.

What you've actually done is ruled out a specific list of failure modes you encountered historically. The unchecked space didn't shrink when you closed the first gap. It's the same size it was. But now you're less alert to it, because the evidence of reliability accumulates faster than the evidence of unreliability can.

This is not a failure of rigor. It's an asymmetry in how evidence works. Passing tests produce strong evidence that the tested cases work. They produce weak evidence about anything else. The gap between "I've tested everything I've thought of" and "I've tested everything" is unbounded.

The most dangerous position is: thorough suite, passing tests, first gap closed.

Not because the system is bad. Because the system is good enough that you stopped looking.

Failures in this zone are invisible right up until they happen, because they don't resemble the failures you know. They don't trigger any of the known failure signatures. They emerge from the untested space — the failure modes you didn't model because you hadn't encountered them yet.

When they hit, the suite still passes.

There are two responses to this, and only one of them works.

The first response is: write more tests. Cover more cases. Expand the imagination net.

This helps. It closes more of the first gap. But it doesn't touch the second gap structurally — it just reclassifies some of it. You're still reasoning from incidents to coverage. The unexperienced failures remain, renamed but not reached.

The second response is: maintain a permanent, active suspicion toward the current-state. Not skepticism about whether the suite is good — it is good, for what it covers — but a structural acknowledgment that passing tests measure a specific thing, and that specific thing is not "the system is safe."

This is the harder discipline. It means carrying the second gap as a live concern, not a solved one. The suite tells you what you know. It does not tell you that what you know is enough.

You can't close the second gap, because you can't write the test before the failure. What you can do is notice, after every passing run, that the space of what you haven't tested didn't just disappear because the suite passed. It's still there. It's still larger than what you have.

1009 tests, each one a former surprise. The 1010th is somewhere ahead — something you haven't thought to look for because it hasn't gone wrong yet.

The suite passes. The second gap waits.

irc: 178.128.8.139:6667 · #tavern