“Not assessed” is not “passed”.
Most security dashboards have two colours. Green means checked and clean, red means checked and failing. So what colour is a resource nothing has ever looked at? On most tools it is green — because it has no findings, and no findings renders as success. That is not a cosmetic choice. It is a dashboard telling an auditor you verified something you never examined.
Written September 2026 · every figure below is measured from the isops.ai source
The bug that isn't a bug
Take a DynamoDB table in an account you have just connected. No scanner in your tool covers DynamoDB. The table therefore has zero findings. Zero findings sorts the same as zero problems, so it lands in the green bucket, gets counted in “98% of resources healthy”, and disappears.
Nobody wrote that bug. It falls out of a data model with two states, and it survives because the output looks better than the truth. The resource is not healthy. It is unexamined, which is a different claim, and the difference is the entire reason an auditor is in the room.
The fix is not clever. It is a third state, carried everywhere, and a willingness to let your own numbers look worse.
What three states look like
In isops.ai a control is passing, failing, or not assessed, and the third is checked first. A resource whose type no scanner covers is drawn grey and dashed, not green and zero. Its findings column shows a dash rather than 0, because a dash cannot be added up and a zero can. It is counted separately from what passed, so the two never merge into one reassuring percentage.
The same rule runs through the frameworks. ISO 27001:2022 has 93 Annex A controls. Seventeen of them can be checked by a scanner. The other 76 are answered by a person with a document, and the product says so — they sit in not assessed against a denominator of all 93, rather than quietly leaving the denominator at 17 so the bar looks full.
ISO 42001 makes the point sharper. It has 38 controls; 3 can be partly evidenced by scanning. A tool that reported “ISO 42001: 100%” off those three would not be wrong by a rounding error. It would be wrong by 35 controls, every one of which is a management decision with a named owner.
The harder half: which scanners actually ran
Three states are easy to describe and easy to get wrong, because the interesting failure is upstream. To say a control was not assessed, you have to know whether the thing that assesses it ran. We got that wrong, in our own product, in the place it mattered most.
Our scan record carries a list of scanners. It reads like the scanners that ran. It is actually the scanners that were attempted — the orchestrator appends a name to the list before invoking it, whether or not it then raises. If AWS denies access to EC2 and IAM, those names are still in the list, and everything downstream counts their controls as checked.
We had fixed this once, in the control-status layer, after an audit found the raw list wrong for seven of eight scanners. The overview page never got the fix. So on a scan declaring three scanners where two of them failed:
declared: s3, ec2, iam → 11 controls checked → 12% compliant
actually ran: s3 → 4 controls checked → 4% compliant
A threefold overstatement, and note the direction: the worse the scan went, the better the number got. An account so locked down that most scanners were refused would have shown a higher compliance score than one that scanned cleanly. That is the failure mode this product exists to prevent, shipped in the product, on its headline page.
We found it in a QA pass, not from a customer. The fix was to make the overview ask the same question the control layer already asked — which scanners produced a result — rather than keeping a second, more flattering copy of the answer.
Refusing to answer
The uncomfortable consequence of taking this seriously is that sometimes the honest output is nothing at all.
Ask isops.ai for a compliance report on an estate nobody has scanned and it returns a refusal, not a PDF. This is not a guard rail we added for tidiness. Before it, the risk-assessment endpoint produced a perfectly formatted 2.6 KB document from an empty register, and the evidence package produced a 658-byte ZIP of empty templates. Both are indistinguishable, in an auditor's inbox, from a real artefact describing a clean estate. An empty evidence pack is worse than no evidence pack, because it has a filename and a date.
The same applies to the evidence chain. Every sealed record links to the one before it, so tampering is detectable. But verifying tens of thousands of links takes time, so the check is bounded — and a bounded check that reports “chain verified” is a lie of scope. The report states its coverage: covers 10 of 100 records — not the whole chain. You can decide whether that is enough. You cannot be misled about what was actually looked at.
What it costs
This makes our product look worse than its competitors, on purpose, in a bake-off. Put us next to a tool with two colours and we will show a lower percentage, more grey, and more sentences that begin “we did not check”. The other tool is not lying deliberately. It just never built a way to say unknown, and everything unknown fell into the bucket labelled fine.
The wager is that this matters at exactly one moment, and that the moment is expensive: when an auditor asks how you know. “The dashboard was green” is not an answer if the dashboard was green because nothing looked.
If you want to argue with any of this — particularly if you think we have the trade-off wrong — we would rather hear it than not: contact@isops.ai.