Issue 01 - September 16, 2026 - 6 min read
The withholding was never the problem. The silence was.
What an empty field on an AI model card should tell you, and why I said so to NIST this week.
This week I filed a public comment with NIST on AI 300-1, the zero draft for public-facing AI documentation, and a companion letter on the AI testing, evaluation, verification, and validation (TEVV) zero draft outline. Both are part of the NIST AI Standards Zero Drafts pilot project. If you run or support a model risk, vendor risk, or data governance function, the argument is yours as much as mine.
The problem in one paragraph
When you validate a third-party model, the provider decides what to share. Training data, equations, model logic, and code are often withheld, and the provider is entitled to withhold them. In my years in second-line validation at a G-SIB, the withholding itself was never what stalled a model. What stalled it was not knowing the shape of the gap. An empty section in vendor documentation looked the same whether the item did not apply, was never tested, or was assessed and held back. Each of those calls for a different compensating control. Finding out which one you were looking at meant a negotiation with each vendor, one gap at a time.
NIST's draft templates have the same blind spot. They govern whether a field gets populated. Nothing tells you what an absent field means.
The fix I proposed
- For any field left empty, the provider declares one of three states: not applicable, not assessed, or withheld
- Where withheld, the provider names a basis: proprietary, security, legal, or privacy
- The declaration discloses the fact of omission, never the content
- The vendor keeps its secrets. You learn the size of the gap on day one
A profile for regulated financial services
The submission also carries a Regulated Financial Services Profile, built on the draft's own Clause 6 mechanism, which adds the fields a validator needs and a provider rarely offers unprompted:
- Evaluation configuration, because the same weights scored 62.7 percent and 99.9 percent on one benchmark under two different harnesses
- Contamination control, because a headline score of 100 percent fell to 39 percent when the test items postdated the training cutoff
- Four attributes of evaluator independence: who commissioned it, who funded it, how long it ran, and whether the evaluator was free to publish
- A change log stating declines, not only improvements
- A named reviewer of record
The TEVV companion
The TEVV letter makes the measurement case for the same fields. A score without its harness does not meet NIST's own definition of a test result. And the word reliability is doing two jobs across the two drafts, one for the measurement and one for the system, which should be separated before both reach ISO/IEC SC 42.
Every regulatory reference in the profile is an informative crosslink. It proposes no mandate, no enforcement, and no legislative vehicle. Both letters were filed in a personal capacity as public comment and are now public record.
Your turn
If you have sat across from a vendor trying to size an undeclared gap, tell me how you handled it. Anonymize everything. I am collecting patterns for a follow-up issue, and if the same three or four gaps show up across a dozen institutions, that is evidence a standards body will listen to.
Joshua M. Reh, Managing Principal, Cijara Group LLC
Both letters were submitted in a personal capacity as public comment and are part of the public record. Nothing in them is endorsed by NIST. Cijara Group holds no federal contract or subcontract related to this work.