AI Vendor Risk Assessment Questionnaire

Can you provide a model card detailing the architecture, known limitations, and benchmark results relevant to our use case?

What foundation models or pre-trained models are used? Are they open-source or proprietary?

What training data sources were used, and how was data quality assured?

Can you explain the model's decision-making process for our specific use case (explainability)?

What is the model's known failure mode profile, and how are edge cases handled?

Is our input data used to train, fine-tune, or improve any shared or public models?

What mechanisms exist to opt out of data being used for model improvement?

How is tenant data isolated in a multi-tenant environment?

Where is our data stored, processed, and backed up? What jurisdictions apply?

What is your data retention policy, and how is data securely deleted upon contract termination?

Do you process any personally identifiable information (PII) through the AI system? If so, what safeguards exist?

What defenses are in place against prompt injection attacks (direct and indirect)?

How do you detect and mitigate model drift over time?

What audit logs exist for model inputs, outputs, and access events?

What is your incident response process for an AI-specific security event (e.g., data leakage via model output)?

How are automated decisions monitored for accuracy and safety?

What penetration testing or red teaming has been performed on the AI system? Provide dates and scope.

How do you manage AI supply chain risks (e.g., compromised model weights, malicious plugins)?

What is your methodology for testing and monitoring bias? Provide recent evidence.

How do you ensure the model does not produce discriminatory outputs across protected characteristics?

What human-in-the-loop mechanisms exist for high-stakes decisions?

Do you have an AI ethics review board or equivalent governance body?

How do you handle misinformation and hallucination risks?

Who holds legal liability for harmful or incorrect model outputs?

Have you completed an EU AI Act conformity assessment (if applicable)?

Are you aligned with NIST AI RMF, ISO/IEC 42001, or equivalent frameworks? Provide evidence.

Have you published an AI-CAIQ (CSA AI Consensus Assessment Initiative Questionnaire) to the STAR Registry?

What contractual commitments exist regarding compliance with evolving AI regulations?

Can you provide a SOC 2 Type II report that specifically covers AI system controls?

What SLAs exist for model uptime, latency, and accuracy?

How are model updates and version changes communicated? Can we test in a sandbox first?

What is the notification lead time before a breaking model change is deployed?

What data portability options exist to avoid vendor lock-in?

What happens to our data and model customizations if we terminate the contract?