Verifying In The Wrong Layer
The check passed in the layer above the one where the error lived.
A verification ran against a rendered summary rather than the underlying data, so it confirmed the summary was internally consistent and said nothing about whether it matched reality. A check has to run in the layer where the failure can actually occur, which is usually one below where it is convenient.
Pattern
You verify a result by inspecting the thing that presents it. The dashboard renders, so the pipeline is fine. The file exists, so the write succeeded. The summary is coherent, so the underlying rows are right.
The error lives one layer below and the check cannot reach it.
Why it looks right
The upper layer is where the work is visible, and inspecting it is fast. It also catches the loudest failures — if the pipeline dies completely, the dashboard is empty and you see it immediately.
That success rate is the trap. The check works for total failures and is blind to partial ones, which are the common kind.
Why it fails
A layer can be internally consistent and externally wrong. A summary computed from stale rows is a perfectly correct summary of stale rows. A file that exists may contain half a write. A rendered page proves the renderer ran, not that the numbers came from where you think.
Worse, each layer tends to normalise what it receives — filling gaps, coercing types, defaulting missing values — so a check above the normalisation sees a tidier world than the one that actually exists. On one pipeline the freshest export day was incomplete by about 42% while the rendered summary above it looked entirely normal.
Instead
Run the check where the failure can occur. In practice this means one layer lower than feels necessary:
comparing revenue across weeks → do not check the chart, check that the export for those dates ran
Three questions make the layer explicit. Where could this specifically go wrong? What is the cheapest observation at that level — a row count, a checksum, a modification time? And does my check distinguish the failure from a normal result, or only from a total outage?
There is a mirrored failure worth naming alongside this one: trusting a tool that checks the right layer but cannot distinguish the failure mode you care about. Same discipline, opposite direction — first establish what a check can see, then decide what its verdict is worth.