Healthcare AI vendor risk · A ReMyll product
Sample assessment Contact
AITPRM Health™A ReMyll product
Evaluation framework

The evaluation framework.

Twelve weighted risk domains, 256 questions, a published scoring rubric and four finding tiers. This page describes how the method works so your committee can inspect it before relying on it.

Request a demoView a sample assessment
Twelve domains

Twelve domains, weighted by how directly a gap exposes the organization.

Coverage never changes with the plan or the reviewer. What changes is depth, and no domain is assessed with fewer than three questions. Weight bands are published rather than tuned per engagement.

D1 · 26 questions · High

Data and model control

Training, tuning, benchmarking and evaluation rights on customer data and derivatives
D2 · 22 questions · High

Subprocessors and supply chain

Named model providers and flow down of PHI restrictions through the chain
D3 · 22 questions · High

Derivative data and embeddings

Ownership, retention and deletion of embeddings, summaries and inferences
D4 · 24 questions · High

Human in the loop controls

Documented human confirmation for decisions affecting care, coverage or billing
D5 · 18 questions · Medium

Suspension and fallback architecture

How the tool is halted, by whom, and what the workflow reverts to
D6 · 23 questions · Medium

Model incident reporting

Notification of drift, bias and hallucination, distinct from data breach
D7 · 21 questions · Medium

Ambient capture governance

Audio ownership, consent model and retention for passive listening
D8 · 19 questions · Medium

Cross border data transfer

Where PHI is stored, processed and backed up, including by subprocessors
D9 · 20 questions · Medium

Cross customer tenant separation

Isolation at the model layer and whether learning from your data reaches other customers
D10 · 22 questions · High

Agentic access and API scope

Permission scope, audit log and rollback for autonomous action in connected systems
D11 · 17 questions · Standard

Termination and data return

Return, destruction and retention of data, prompts, outputs and derivatives
D12 · 22 questions · Standard

Audit rights and inspectability

Direct inspection of the AI system by you or your independent auditor
What this means for you

You never choose which risks to look at. All twelve domains are assessed on every vendor; the plan and the deployment pattern set how deep each goes.

Scoping

Six facts decide which questions apply.

Clinical context, systems touched, write access, autonomy level, states of operation and the decision date resolve to one of thirteen deployment patterns. The pattern sets question depth; state overlays add jurisdiction-specific questions. A risk floor means PHI, subprocessor and termination questions always apply.

Core sets range from roughly 90 to 180 questions depending on pattern
State overlays add questions by deployment location
Vendors see only their scoped set
Scope is recorded with the assessment and cannot be edited after the vendor is invited
What this means for you

You answer six facts about how the tool will be used. The platform selects the questions and the state overlays; a risk floor means nothing critical can be scoped away.

Scoring

A published rubric, with partial credit.

Every question carries a scoring band from 0 to 5. Answers are scored against the band; evidence is rated separately as verified, partial, unverified, documentation gap or missing. A reviewer note is required wherever the applied score departs from the suggested score. Domain scores are weighted into the overall score out of 100.

5Obligation assumed in writing, with current verified evidence
4Obligation assumed in writing; evidence partial or pending
3Practice described credibly; not yet contractual
2Practice described in part; material gaps
1Answer does not address the question or defers to a future state
0Not answered, or response not substantiated by submitted evidence
What this means for you

A score you can explain to a committee. Every applied band is visible, every departure from the suggested band carries a reviewer note, and an open finding caps the total.

Findings

Four severity tiers. Tier 4 pauses the review pending resolution.

A gap between what was asked and what was shown becomes a finding with a tier, a domain and an owner. Open findings cap the overall score. A tier-4 finding pauses the review until the item is resolved or formally accepted by the committee.

1Documentation gapMissing or outdated artifact for an otherwise adequate answer. No score cap.
2Control weaknessPractice exists but is informal or unenforced. Caps the domain score.
3Contractual gapMaterial obligation not assumed in writing. Caps the overall score; generates a sample contract control.
4Documentation gapSubmitted evidence does not substantiate the response, or a control requires vendor-side action. Domain score is held at a ceiling pending verified documentation.
What this means for you

Findings are what you act on. Each carries a tier, an owner and a due date, and qualifying findings generate the contract control that closes them.

Sample questions

Six questions in full.

Every question names the provision behind it, the evidence artifact it expects in return, and the logic used to score the response. Six are shown in full so your team can judge the method rather than take our description of it.

aitprm.com/library/questions
Questions you send to your vendorVerbatim
Q 01D1
Training rights and derivation
D1 · HIPAA 164.502(b) · 45 CFR 164.504(e)

Does the vendor train, tune, benchmark or evaluate any model on our data, including prompts, outputs, embeddings or other derivative artifacts?

EvidenceAn executed agreement plus a zero data retention addendum naming prompts, outputs, embeddings and tuned parameters. A policy page does not satisfy this.Scoring100 where the addendum is executed and names derivatives. 70 where contractual but silent on derivatives. 40 where policy only. 0 where training on customer data is permitted.
Q 02D2
Subprocessor chain of custody
D2 · HIPAA 164.308(b)

List every subprocessor in the chain that can access clinical content, the country each operates from, and the agreement status at each hop to the foundation model API.

EvidenceA multi hop architecture map, the subprocessor register with country and agreement status, and one executed downstream agreement as proof of practice.Scoring100 where the full chain is mapped with agreements at every hop. 70 where mapped with one gap. 40 where only direct subprocessors are named. 0 where the vendor cannot enumerate its chain.
Q 03D10
Agentic write back limits
D10 · NIST AI RMF MANAGE 2.2

If the system writes into the clinical record or submits transactions, what function calling scopes are granted, does a person confirm each write, and what is the rollback path?

EvidenceA permission matrix of granted scopes per connected system, a sample immutable audit log entry for a write action, and the documented rollback procedure with test evidence.Scoring100 where scopes are least privilege, writes are confirmed and rollback is tested. 70 where confirmed but rollback untested. 40 where logged but unconfirmed. 0 where autonomous write back has no human checkpoint.
Q 04D4
Human control over determinations
D4 · CMS · California SB 1120

Where output affects care, coverage or billing, what is the documented human control and how is it enforced contractually?

EvidenceWorkflow documentation plus the contract clause requiring individualized review by credentialed personnel.Scoring100 where review is contractual and evidenced in workflow. 70 where documented in workflow only. 40 where asserted. 0 where output can stand alone.
Q 05D5
Suspension without vendor action
D5 · NIST AI RMF MANAGE 2.4

Can we suspend the system without vendor action, how quickly does that propagate, and when was it last tested?

EvidenceThe documented suspension procedure, the propagation time, and dated evidence of the last activation test.Scoring100 where customer held and tested within 12 months. 70 where customer held but untested. 40 where vendor assisted. 0 where suspension requires a support ticket.
Q 06D6
Model incident notification
D6 · HITECH · OCR

What is the notification timeline for drift, bias or hallucination with clinical impact, distinct from a data breach?

EvidenceThe incident response policy showing AI failures defined separately from security incidents with their own clock.Scoring100 where AI incidents carry a defined contractual window. 70 where defined in policy. 40 where handled under breach terms. 0 where undefined.
Scoring bands shown as published. Evidence is rated separately as verified, partial, unverified, documentation gap or missing.
Regulatory references last reviewed September 2026.See the full question set on a demo →
Sources

Every question records where the requirement came from.

Obligations arise from statute, regulation, agency guidance, enforcement practice, contractual terms, accreditation requirements and recognized frameworks. Each question records its source so the record shows why it was asked.

NIST AI RMF 1.0 (mapped)
ISO/IEC 42001 (aligned)
HIPAA and HITECH, including 45 CFR 164.504(e)
42 CFR Part 2
ONC HTI-1
CMS Medicare Advantage rulemaking
FDA clinical decision support guidance
OCR enforcement practice
State AI statutes: Texas, California, Colorado, Utah, Illinois, Arizona
Next step

See the method applied to one of your deployments.

We scope one of your AI tools live and walk through the questions it would receive.