"Zero Problems Found" Means Zero Of What You Measure

A freshness monitor reported all clear while the pipeline behind it was failing. The monitor was right about the only thing it could see.

Pattern

A monitoring script reports 0 problem sources. You repeat that upstream as "the pipeline is fine, the problem only exists in stale notes." Later someone sends a screenshot of the actual job failing with a database read error after 12 minutes.

Why it looks right

The tool ran. It produced a clean number. It was written for exactly this purpose, and it was not lying — it answered its own question correctly.

The trap is that the question was narrower than the answer sounded. The script compared file timestamps: did a file arrive in the last 7 days? Against that question, a crashed export and a source that was deliberately retired look identical. Both are simply absent.

Why it fails

A check has a set of failure modes it can distinguish and a much larger set it cannot. 0 problems always means zero of the things I am able to see. When that sentence gets shortened to 0 problems and repeated by something that did not write the check, the caveat is gone and the confidence is not.

This is the same family as verifying in the wrong layer, but pointing the other way: there the measurement used the wrong instrument, here a correct instrument was trusted beyond its range.

Instead

Before quoting a tool's verdict, answer one question: which failure modes can this check not distinguish?

0 problem sources — timestamp comparison only; cannot distinguish a crashed job from a retired one.

Two habits follow. Make tools name their own scope in their output, so the caveat travels with the number. And when a verdict contradicts a human report, assume the tool is narrow before assuming the human is wrong — the human saw an error message, the script saw a file date.