Making the evidence behind AI decisions trustworthy enough to govern.
Decisions about AI — deployment approvals, risk acceptances, regulatory certifications — are increasingly evidence-dependent. Yet the artefacts that today's evaluation practice produces are often insufficient, not credible, or not auditable for the governance purposes they are meant to support. A model can pass every internal benchmark and still fail the questions a board, an auditor, or a regulator will actually ask.
This paper introduces evaluability: the capacity to produce and validate decision-ready evidence for AI governance. Evaluability turns on a single operational question — does the available evidence satisfy the requirements the governance decision imposes? — and on one principle that reorganises everything downstream: evidence is never sufficient in the abstract. It is sufficient only relative to the decision it must support.
This is a position paper. It establishes the why and the what. Operational methods, metrics, and tools are implementations of this framework and are treated as such — not as the framework itself.
Over the past decade, organisations have invested heavily in model capability, and evaluation has matured alongside it: standardised benchmarks, automated scoring pipelines, model-graded judging, red-teaming protocols. By most measures we are better at measuring AI than ever before.
And yet governance failures persist. Models pass internal evaluation and fail in production. Systems clear a benchmark and stumble under regulatory scrutiny. High scores prove brittle under distribution shift. Teams find they cannot justify a deployment to the people accountable for it. The uncomfortable question is why better evaluation has not produced better governance.
A score is a snapshot. Evidence is a chain.
The answer is that evaluation and governance deal in different currencies. Evaluation produces measurements — a fact about a test set, at a point in time, under controlled conditions. Governance consumes evidence — a structured body of information that justifies a claim about a system's fitness for a specific purpose, at a specific level of stakes, in a form that can withstand review. A 94% accuracy figure is a measurement. It becomes evidence only once someone can show that the test set resembles deployment, that it was not contaminated, that the metric fits the use, that the result is reproducible, and that the uncertainty around it is understood.
We have operationalised the production of measurements. We have not operationalised the production of governance-ready evidence. The gap is not primarily technical — it is conceptual. We lack a shared account of what counts as evidence for an AI governance decision, and a way to assess whether what we produced is fit for that decision. Evaluability is that account.
If we cannot say what evidence is, we cannot assess its quality. Governance evidence is not a single kind of thing; it comes in five types, and the distinctions matter for what a decision-maker should ask.
| Type | What it is | Example |
|---|---|---|
| Direct | Measurement on the target deployment task itself | Performance on real production traffic |
| Indirect | Measurement on a proxy or benchmark | A public benchmark or internal proxy task |
| Supporting | Context that lets the measurement be interpreted | Lineage, versioning, dataset documentation |
| Meta-evidence | Evidence about the evidence | A judge-reliability study; a contamination screen |
| Absence | Documentation of what was not tested, and why | Explicit statement of omitted scenarios |
A benchmark score is indirect evidence: it supports a claim about capability only if a supporting argument shows the benchmark represents the deployment. Absence evidence is often the decisive kind in a regulated setting — a reviewer asks not only what you tested but what you did not, and why. The omission of a failure-mode analysis is itself a governance signal. Evaluability assesses evidence of all five types against the requirements of the decision, not just the headline number.
Sometimes evaluation produces evidence. Often it produces measurements that are treated as if they were evidence. That substitution is a category error, and it is the root of a large share of governance failures.
| Evaluation | Evidence |
|---|---|
| Measures performance | Justifies a governance claim |
| Answers: what is the score? | Answers: why should we believe the score is enough? |
| A snapshot at a point in time | An argument linking measurement to decision |
| Produces observations | Produces defensible claims |
The dangerous assumption is that good evaluation results mean good evidence. A strong score on a contaminated benchmark, graded by an unvalidated judge, on a test suite that never touched the hard cases, is not moderately good evidence — it is a well-documented artefact of the wrong measurement. Evaluation measures models. Governance requires evidence about models. These are different activities, with different standards and different outputs.
Three properties determine whether evidence is trustworthy enough to act on. They are independent: each catches a distinct class of failure, and a shortfall in one is not offset by strength in another.
Is the evidence complete enough for this decision?
Does it cover the operating envelope, span the plausible failure modes, and address every claim the decision rests on?
Can we believe how it was produced?
Were the datasets representative and uncontaminated, the judges validated, the metrics fit, the methods independent and documented?
Can an independent party verify it?
Is the evidence versioned, traceable, and preserved so a reviewer can reconstruct the result rather than take it on trust?
Evidence is never sufficient in the abstract. It is sufficient only for a decision.
This is why trustworthy enough is a relative standard, not an absolute one. Evidence sufficient for internal experimentation may be wholly insufficient for regulatory approval; a test set that suffices for a low-stakes assistant will not suffice for a credit or clinical decision. The standard is set by the decision's materiality — the weight of its consequences.
Evidence is trustworthy enough when it is sufficient, credible, and auditable relative to the specific governance decision and its materiality — no more, and no less, than the decision requires.
If sufficiency, credibility, and auditability define decision-ready evidence, what evidence does a particular decision require? Generic checklists cannot answer this. Requirements are derived from four features of the decision.
What is being decided shapes what must be shown. An approval demands pre-deployment coverage of the envelope; a continuation demands monitoring evidence that behaviour has not drifted from what was certified; a retirement demands something different again. The objective fixes the claims that must be supported.
The stakes set the strength. Low-materiality tools warrant light requirements; safety-, credit-, or clinically-critical deployments warrant stringent ones, including independent validation and continuous monitoring.
The conditions of use define what coverage means — the data distributions the system will meet, its edge cases, its stress regimes, its failure and fallback conditions.
A governance claim breaks into sub-claims, and each sub-claim generates evidence requirements. "This system is fair" decomposes across groups and their intersections; "this assistant is suitable for support" decomposes into grounding, retrieval accuracy, and refusal behaviour. Each leaf of the decomposition names a dimension that evidence must address.
Requirements can only be derived against a bounded operating domain. A deployment whose admissible inputs, outputs, and authority are unbounded cannot be comprehensively evidenced by any finite evaluation — it is unevaluable by construction, and the honest governance response is to bound its scope, not to lower the bar. Bounding is a design lever: retrieval grounding, output schemas, and authority scoping each convert part of the unevaluable into the evaluable.
Deriving what evidence a claim requires is not new. The assurance-case tradition — Claims–Arguments–Evidence and Goal Structuring Notation — has structured exactly this derivation in safety-critical engineering for decades, and model risk management (the SR 11-7 lineage) has long required independent validation, effective challenge, and tiered scrutiny. Evaluability does not replace these methods; it begins where they end.
Assurance cases define what evidence a claim requires. Evaluability asks whether the evidence produced actually meets that standard.
In practice, organisations beginning an evaluability assessment often discover that requirements were never derived. There are evaluation results, but no link to the decision; benchmarks, but no articulation of what sufficiency would mean; metrics, but no auditable justification for the choices. This absence is itself the finding. The most common evidence failure is not weak evidence against a stated requirement — it is evidence collected against no requirement at all.
Evaluability is the capacity to produce and validate decision-ready evidence — evidence sufficient, credible, and auditable enough to support a governance decision throughout an AI system's lifecycle.
The capacity has two levels, and both must hold. A system is evaluable when its operating domain is bounded and instrumented such that decision-ready evidence about its behaviour can, in principle, be produced. An organisation has evaluability when it can generate that evidence, independently validate it, and act on it — determining at any point whether a deployment remains aligned with its purpose, and intervening when it does not.
Two mechanisms make the capacity concrete. Evaluation quality is the property of the evidence itself — the degree to which it is sufficient, credible, and auditable. Evidence validation is the function that assesses evaluation quality: the same effective challenge that model validation brings to models, applied one level up, to the evaluation and the evidence it produces.
In practice, evaluability reduces to one question asked of every governance decision:
Does the available evidence satisfy the requirements the governance decision imposes?
And crucially, the answer is consequential. Where the evidence falls short of what the decision's materiality requires, the decision resolves toward the stricter outcome — pending stronger evidence — rather than inheriting the treatment its headline score would suggest. Weak evidence is meant to be expensive. That is what turns evaluability from a reporting exercise into a governance control, and what first creates an economic incentive to invest in evidence, not only in the model.
Evaluability is a governance lens. It asks whether the whole argument — from claim to evidence to decision — is complete, credible, and sustainable, and it makes the answer change what the organisation is allowed to do.
The following illustration is constructed, but every failure mode in it is drawn from documented behaviour of real evaluation pipelines — benchmark contamination, model-judge miscalibration, and coverage mismatch.
A bank deploys a retrieval-grounded assistant that answers employees' HR-policy questions, and reports 94% accuracy at validation: a thousand generated test questions, of which a model-based judge passed 940. On its face, a strong result. Evaluability asks not whether 94% is high, but what decision it justifies.
Three findings emerge. First, coverage: review shows 88% of the generated questions are single-document lookups, while multi-document synthesis, stale-policy conflicts, and ambiguous entitlement questions — precisely the queries that drive HR escalations — are nearly absent. The 94% describes the easy region. Second, credibility: the judge was never validated against human labels. A 200-item human-labelled sample later shows it agrees with people 96% of the time on lookups but only 71% on synthesis, systematically crediting fluent-but-ungrounded answers exactly where the deployment is hardest. Third, auditability: the judge prompt was revised mid-evaluation without versioning, so the headline figure cannot be reproduced.
Once the evidence is corrected for the judge's error and re-weighted to the mixture of questions production actually sees, the defensible accuracy on the queries that matter falls well below the headline. Nothing about the model changed; everything about the decision did.
The governance outcome is not "reject." It is bounded: deploy autonomously on single-document lookups, where the evidence is strong; route synthesis and entitlement questions to human review; refuse and escalate policy-conflict queries, for which no evidence exists at all. The headline number never changed — but what it is allowed to license is now derived, not asserted. This bounded outcome is the characteristic product of an evaluability assessment.
Evaluability is a synthesis, and it is careful about what it claims to be new. Several mature traditions already govern parts of this territory, and evaluability is designed to sit alongside them.
What has changed is the evidence chain itself. Where evaluation once meant a held-out set and a computable metric, it is now a pipeline: tests generated by models, correctness judged by models, benchmarks that leak into training data, input spaces that are unbounded. Each link is itself a model with its own failure modes, and almost none of it receives the validation routinely applied to the deployed model it certifies. Evaluability's contribution is to extend effective challenge from the model to this modern evidence chain, and to make the result consequential. It does not claim that no one has ever scrutinised evidence — regulated industries have done so for decades. It claims that the scrutiny has not yet caught up to how AI evidence is now produced.
Boards are increasingly accountable for AI risk and need to know the evidence put before them is decision-ready. Evaluability supplies the questions: What is the evidence this system is fit to deploy? How do we know it is sufficient for this decision? Who independently reviewed it? How is it maintained over time?
The EU AI Act's requirements for documentation, conformity assessment, and post-market monitoring are, in substance, evidence requirements. Sufficiency, credibility, and auditability give those obligations a conceptual spine — and a shared vocabulary for what an evaluation does and does not certify.
Validation frameworks already require independent review and ongoing monitoring. Evaluability extends them to the evidence behind the model decision: not only whether the model is validated, but whether the validation itself produced evidence that is credible and auditable.
These are where the evidence chain is longest and the gate bites hardest. For a retrieval system, evaluability asks whether the evidence for grounding and faithfulness is sufficient, credible, and auditable. For an agent, it asks the same of trajectory safety, authority alignment, and contingency handling — over sequences and environments, not single responses. Because the framework attaches to the decision rather than the architecture, it scales to each.
This paper is deliberately conceptual. Making evaluability operational requires instruments — ways to score evaluation quality, size the human labels a credible judge validation needs, screen benchmarks for contamination, and quantify how far a headline should be discounted when the evidence is weak. Those instruments exist and are developing quickly, but they are implementations of this framework, not the framework itself. The framework stands independently of any single tool or metric, and any such tool should be judged by whether it serves the question at the centre of this paper.
Several questions define the road ahead: whether sufficiency, credibility, and auditability can be formalised end-to-end; what it means for evidence to be complete, and whether completeness can be measured; how organisations accumulate poor evidence over time — an evidence debt — and how it can be remediated; and how evidence requirements should adapt as systems become more autonomous and their behaviour more dynamic. If evaluability matures into a recognised discipline, these questions will be its frontier — and that recognition is something the field grants through use, not something a framework can award itself.