This is a ready-to-use credibility assessment report. It is the reasoned argument that a model’s output can be trusted for a stated use, sized to the risk, and it is the document an inspector or a reviewer reads to understand why you rely on the model. It complements, and does not replace, the model card factsheet and the underlying validation records. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and route it through your normal document control, review, and approval. A worked filled specimen follows the template so you can see how a completed version reads. AI-specific guidance is still evolving, so confirm each cited reference against the current source before you rely on it, and treat any draft guidance as draft.
Document control header
| Field | Entry |
|---|---|
| Document title | AI Model Credibility Assessment: <<FILL: MODEL NAME>> |
| Document number | <<FILL: DOC-ID, e.g. CRED-AI-007>> |
| Version | <<FILL: version, e.g. 1.0>> |
| Effective date | <<FILL: effective date>> |
| Supersedes | <<FILL: prior version or "New">> |
| Model name and version | <<FILL: model name, version, and the exact model artifact hash or ID>> |
| Associated model card | <<FILL: model card doc number>> |
| Associated risk assessment | <<FILL: AI risk assessment doc number>> |
| Author | <<FILL: role, e.g. Data Science / Validation lead>> |
| Assessment owner | <<FILL: accountable role, e.g. Digital Quality lead>> |
1. Purpose
This report documents the credibility assessment for <<FILL: MODEL NAME>> so that reliance on its output for <<FILL: the stated use>> is supported by evidence sized to the risk. Credibility here means the degree of trust that can be placed in the model’s output for a defined context of use, established by evidence rather than asserted. The report follows a fixed sequence: state the context of use, size the model risk, plan the evidence, gather it, then judge whether it is adequate.
2. Context of use (Step 1)
State the use in one sentence that names the output, the decision it feeds, and the accountable role. Everything downstream is scoped to this statement.
| Field | Entry |
|---|---|
| Model output | <<FILL: exactly what the model produces, e.g. a proposed complaint category>> |
| Decision the output feeds | <<FILL: the decision, e.g. initial complaint routing>> |
| Accountable human role | <<FILL: who owns the final decision, e.g. intake analyst>> |
| Conditions and boundaries | <<FILL: input types, populations, sites, and limits within which the model operates>> |
| Explicitly out of scope | <<FILL: uses this assessment does NOT cover>> |
| One-sentence context of use | <<FILL: "The model does X, which feeds decision Y, subject to Z by role R.">> |
If the context of use cannot be written in one clean sentence, the model is not ready to be assessed. Stop here and pin it first.
3. Model risk (Step 2)
Model risk sets how much evidence is required. Score it from two factors and combine them.
| Factor | Definition | This model |
|---|---|---|
| Model influence | How much the output drives the decision, relative to other evidence and human judgment | <<FILL: Low / Medium / High + one-line basis>> |
| Decision consequence | Severity of the outcome if the decision based on the output is wrong | <<FILL: Low / Medium / High + one-line basis>> |
| Combined model risk | Influence weighed against consequence | <<FILL: resulting risk level>> |
Use the grid to translate the combined risk into an evidence tier.
| Low decision consequence | High decision consequence | |
|---|---|---|
| Low model influence | Minimal evidence: basic performance on a held-out set, human review defined | Moderate evidence: representative test set, error-type analysis, monitoring, strong human review |
| High model influence | Moderate evidence: representative test set, calibration, monitoring | Maximal evidence: rigorous held-out testing with uncertainty, failure-mode analysis, independent controls, continuous monitoring |
| Field | Entry |
|---|---|
| Evidence tier selected | <<FILL: Minimal / Moderate / Maximal>> |
| Rationale | <<FILL: why this tier follows from the influence and consequence above>> |
4. Credibility plan (Step 3): evidence planned before it was gathered
Record the plan and its date. A plan written after the results is rationalization, and the dates give it away.
| Planned evidence element | Specification | Acceptance threshold |
|---|---|---|
| Test population and hold-out method | <<FILL: source, time split, how held out from training and tuning>> | <<FILL: representativeness criterion>> |
| Primary metrics | <<FILL: e.g. per-class recall/precision, not accuracy alone>> | <<FILL: justified thresholds, tied to error consequence>> |
| Uncertainty treatment | <<FILL: confidence intervals, sensitivity, sample-size floor>> | <<FILL: e.g. interval width acceptable at n>> |
| Calibration | <<FILL: method>> | <<FILL: acceptance>> |
| Subgroup reporting | <<FILL: subgroups that matter: product, site, rare class>> | <<FILL: minimum per-subgroup performance>> |
| Explainability evidence | <<FILL: if any, and what it does and does not claim>> | <<FILL: acceptance>> |
| Human control | <<FILL: what a person reviews, against what, where recorded>> | <<FILL: control is defined and tested>> |
| Monitoring | <<FILL: metrics, thresholds, cadence>> | <<FILL: live before go-live>> |
| Plan date | <<FILL: date the plan was approved, before evidence generation>> |
5. Evidence gathered (Step 4)
Report the results against the plan. Attach the underlying validation and test records; do not restate them in full here.
| Evidence element | Result | Meets threshold? |
|---|---|---|
| Held-out performance (per subgroup) | <<FILL: results with uncertainty>> | <<FILL: Yes / No>> |
| Metrics vs consequence | <<FILL: recall on the categories where a miss matters, etc.>> | <<FILL: Yes / No>> |
| Calibration | <<FILL: result>> | <<FILL: Yes / No>> |
| Weak spots identified | <<FILL: subgroups or cases that underperform>> | n/a |
| Human control verified | <<FILL: how the review step was confirmed and recorded>> | <<FILL: Yes / No>> |
| Monitoring live | <<FILL: metrics and starting values>> | <<FILL: Yes / No>> |
6. Adequacy judgment (Step 5)
This is the judgment the whole report exists to support. It cannot be replaced by a formula.
| Field | Entry |
|---|---|
| Does the evidence justify the trust the context of use requires? | <<FILL: Adequate / Adequate with conditions / Not adequate>> |
| Conditions or bounds | <<FILL: e.g. "adequate as a reviewed suggestion; two low-volume categories on watch">> |
| If not adequate, the chosen path | <<FILL: gather more evidence / reduce influence via stronger human control / narrow the context of use>> |
| Watch items fed to monitoring | <<FILL: the weak spots to track and re-evaluate>> |
| Re-assessment trigger | <<FILL: events that force a fresh credibility assessment: model change, drift breach, use change>> |
7. Acceptance criteria
The assessment is acceptable when all of the following hold:
- The context of use is a single sentence naming output, decision, and accountable role.
- Model risk is scored from influence and consequence, and the evidence tier follows from it.
- The credibility plan is dated before the evidence was generated.
- Evidence is reported per relevant subgroup, with uncertainty, against thresholds tied to error consequence.
- A human control is defined and verified where the consequence is real.
- Monitoring is live before reliance begins.
- The adequacy judgment is explicit, signed, and honestly bounded, not a blanket trust claim.
8. References
ASME V&V 40, Assessing Credibility of Computational Models (the context-of-use and model-risk framing). FDA draft guidance, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products (January 2025 draft; verify current status and scope). ICH Q9(R1), Quality Risk Management (risk-based evidence sizing). FDA and EMA, Guiding Principles of Good AI Practice in Drug Development (14 January 2026), and joint FDA/Health Canada/MHRA transparency principles for machine-learning-enabled devices. 21 CFR Part 11 and EU GMP Annex 11 for the records the model produces and consumes. EU AI Act, Regulation (EU) 2024/1689, for high-risk system obligations where applicable.
Confirm the current version and status of each reference before issue; AI-specific guidance is moving quickly.
9. Revision history
| Version | Date | Author | Summary of change |
|---|---|---|---|
<<FILL: 1.0>> | <<FILL: date>> | <<FILL: author>> | Initial issue. |
10. Approvals
| Role | Name | Signature | Date |
|---|---|---|---|
| Author (data science / validation) | <<FILL>> | ||
| Model / process owner | <<FILL>> | ||
| Quality Assurance (adequacy judgment) | <<FILL>> |
Filled specimen
The following shows the core of a completed assessment for an example model that screens incoming product complaints and proposes a category for an analyst to confirm. The company, numbers, and results are illustrative; replace them with your own.
Context of use. The model proposes one of twelve complaint categories for each new complaint; the intake analyst confirms or changes the category before routing; the analyst owns the final category. One sentence: “The model suggests a complaint category, which sets initial routing, subject to analyst confirmation before any effect.”
Model risk. Influence: Low to Medium (every output is reviewed by a person before it has any effect). Consequence: Medium (a misroute the analyst fails to catch could delay a safety-relevant complaint). Combined: Moderate. Evidence tier: Moderate.
Credibility plan (approved 03 March 2026, before evidence). Hold out a test set from a later quarter than training, unseen in training or tuning. Report per-category recall and precision with 95 percent confidence intervals. Threshold: safety-relevant categories must reach recall of at least 0.95, accepting lower precision there; routine categories at least 0.85 recall. Confirm the analyst-review step is defined and recorded. Stand up monitoring on override rate and input distribution from day one.
Evidence gathered.
| Element | Result | Meets threshold? |
|---|---|---|
| Safety-relevant category recall | 0.97 (95% CI 0.93 to 0.99) | Yes |
| Routine category recall (10 of 12) | 0.86 to 0.94 | Yes |
| Two low-volume categories | recall 0.71 and 0.74 | No, flagged |
| Calibration | acceptable (reliability curve within band) | Yes |
| Human control | analyst confirmation recorded in the complaint record with model version | Yes |
| Monitoring | override rate 6%, input distribution baselined | Yes, live |
Adequacy judgment. Adequate with conditions: trustworthy as a reviewed suggestion across ten of twelve categories; the two low-volume categories are on watch in monitoring and re-evaluated at the next quarterly review. Re-assessment triggers: any model retraining, an override-rate breach above 15 percent, or any change to the context of use. Signed by Head of Digital Quality, 18 March 2026.
What made it credible was not a headline score. It was a tight context of use, evidence sized to the risk, honest treatment of the two weak categories, a real human control, and live monitoring. The same model dropped into unreviewed auto-routing would carry far higher influence and would not be credible on this evidence.
Common inspection findings this report prevents
- Reliance on a model with no written context of use, so the evidence was never sized to anything.
- A strong headline accuracy figure with no held-out test, no subgroup breakdown, and no uncertainty.
- A credibility plan dated after the results, so the standard was fitted to the outcome.
- An adequacy conclusion of “the model is trustworthy” with no bounds, no watch items, and no human control.
- A model that was credible at release but has no monitoring, so the claim is a historical one.
How to adapt this report
- Set your document number, owner, and effective date in the header, and link the model card and risk assessment.
- Write the context of use first and get it to one sentence before touching the risk grid.
- Size the evidence tier from influence and consequence, and let that tier drive the plan, not the other way round.
- Date the credibility plan before you generate any evidence, and keep that date visible.
- Report performance by the subgroups that matter for your use, with uncertainty, never as a single aggregate.
- State the adequacy judgment as an explicit, bounded conclusion, and feed every watch item into monitoring.