Sample question 1 · AI Auditing Tools and Techniques (21%)
Meridian Bancorp, a mid-sized U.S. retail bank, has deployed a generative AI chatbot called 'ClarityBot' through a third-party vendor, NovaMind AI, to handle customer loan inquiries. During audit planning, the AI audit team discovers that NovaMind AI provides monthly performance reports to Meridian's IT department, but these reports only include uptime metrics and average response times. The audit team wants to use AI-assisted data analytics to enhance the audit execution phase by ingesting ClarityBot's interaction logs directly. However, the vendor contract specifies that raw interaction logs are the intellectual property of NovaMind AI and may only be shared in aggregated, anonymized form. The audit team has been given access to six months of aggregated output summaries but cannot access the underlying prompt-response pairs needed to assess response accuracy, hallucination rates, or demographic bias patterns. As the AI audit lead at Meridian Bancorp, which action should you recommend first to address the data access limitation and enable meaningful AI-assisted audit analytics?
- A. Proceed with the aggregated summaries provided by NovaMind AI, supplementing the analysis with customer complaint records and regulatory correspondence to infer potential bias or accuracy gaps without requiring raw log access.
- B. Escalate to Meridian's legal and procurement teams to invoke or renegotiate audit rights clauses in the NovaMind AI contract, specifically to obtain access to a statistically representative, privacy-compliant sample of prompt-response interaction logs necessary for audit evidence.
- C. Engage NovaMind AI directly to request that they conduct their own internal audit of ClarityBot's accuracy and bias metrics and provide the resulting attestation report to Meridian's audit team as a substitute for direct log access.
- D. Recommend that Meridian suspend ClarityBot operations until the vendor agrees to provide full unrestricted access to all raw interaction logs, citing the inability to audit the system as an unacceptable operational risk.
View answer & explanation
Correct answer: B. Escalate to Meridian's legal and procurement teams to invoke or renegotiate audit rights clauses in the NovaMind AI contract, specifically to obtain access to a statistically representative, privacy-compliant sample of prompt-response interaction logs necessary for audit evidence.
[Domain 3 - AI Auditing Tools and Techniques, Subtopic 3D - Audit Data Quality and Data Analytics - Area 1 - Audit Data Quality and Data Analytics] Core Concept: Effective AI-assisted audit analytics depend on access to sufficiently granular, representative data. When vendor contracts restrict access to raw operational data, the audit team must work through formal contractual and legal channels to secure appropriate audit rights, as aggregated summaries alone are insufficient to assess model behavior, bias, or accuracy at the level required by AI governance standards. Correct Answer (B): Invoking or renegotiating audit rights clauses is the correct first step because it addresses the root cause — a contractual gap — through the appropriate organizational channel (legal and procurement). Leading practice under ISACA AI audit guidance and the NIST AI RMF requires that organizations retain the right to audit third-party AI systems, including access to representative interaction data needed to evaluate fairness, accuracy, and hallucination risk. Correct Answer (B): Correct Answer (B): This action enables the audit team to obtain the evidence quality necessary for meaningful AI-assisted analytics. Why the other options are incorrect: Why the other options are incorrect: Why the other options are incorrect: Option A: : : Relying solely on aggregated summaries and complaint records is an indirect, compensating approach that cannot substitute for direct evidence of model output quality; it may miss systematic bias or hallucination patterns not reflected in customer complaints, leaving significant audit gaps. Option C: : : Accepting a vendor-conducted self-attestation as a substitute for independent audit evidence violates the independence principle of auditing; the audit team cannot rely on the auditee's own assessment without independent corroboration or direct evidence access. Option D: : : Recommending immediate suspension of ClarityBot is a disproportionate response at this stage; the audit team should first exhaust contractual and legal remedies to obtain necessary data access before escalating to operational disruption recommendations.