What are the risks of generative AI?
By Nick Jang, founder of MINERVA Studio ·
The short answer
The best-known list of generative AI risks comes from NIST's Generative AI Profile (NIST AI 600-1), published on 26 July 2024 as a companion to the AI Risk Management Framework. It names twelve risks that generative AI creates or makes worse: CBRN information or capabilities; confabulation; dangerous, violent or hateful content; data privacy; environmental impacts; harmful bias or homogenization; human-AI configuration; information integrity; information security; intellectual property; obscene, degrading or abusive content; and value chain and component integration. For a bank or insurer the ones that bite first are usually confabulation, data privacy, information security, human over-reliance and the risks that come in with a vendor's model.
NIST's twelve risks, in plain words
| Risk | What it means | In a bank or insurer, for example |
|---|---|---|
| Confabulation | Confident, wrong output, often called hallucination. | A chatbot quotes a fee or a policy term that doesn't exist. |
| Data privacy | Personal data leaked, misused or re-identified. | Customer details pasted into a public chatbot, or surfacing in another user's answer. |
| Information security | Easier attacks, and new attacks on the AI itself. | Prompt injection through an uploaded document; AI-written phishing aimed at staff. |
| Human-AI configuration | People trusting the AI too much or too little, or not knowing it's AI. | An underwriter accepts a summary without checking the source file. |
| Harmful bias or homogenization | Unfair outputs, and everyone getting the same answer. | Marketing copy or decisions that treat groups differently. |
| Information integrity | Making convincing false content cheap. | Deepfake voice used to authorize a payment. |
| Intellectual property | Reproducing protected material. | Generated text or code that copies a licensed source. |
| Value chain and component integration | Untraceable models, data and components from third parties. | A vendor quietly swaps the model behind its product. |
| Dangerous, violent or hateful content | Harmful content produced or made easier to get. | A customer-facing assistant pushed into abusive replies. |
| Obscene, degrading or abusive content | Including synthetic sexual imagery. | Misuse of an image tool on a corporate account. |
| CBRN information or capabilities | Easier access to weapons-related information. | Rarely relevant to banking use, but in the general list. |
| Environmental impacts | Energy and resources used to train and run models. | Part of climate and sustainability reporting. |
NIST groups them three ways: risks in the technology itself, risks from misuse by people, and risks to the wider ecosystem and society. Several cross over.
Why generative AI is different
A credit model gives a score from fixed inputs. A generative model writes new text, code or images from whatever it's given, so its outputs can't all be tested in advance, the same question can get different answers, and the input can carry instructions. That's why prompt injection and confabulation dominate the risk lists, and why testing has to include adversarial attempts, not just accuracy on a fixed sample.
It also arrives through more doors: tools the bank builds, AI features switched on inside software it already licenses, and staff using public chatbots. The last two are where shadow AI comes from.
Controls that reduce the risks
- Know where it's used. An AI use case inventory, vendor features included.
- Set the rules for staff in an acceptable use policy: what data may go in, which tools are approved.
- Ground the answers in the institution's own documents, and check that outputs are supported by them.
- Put [guardrails](/answers/ai-guardrails) around inputs and outputs: sensitive data filters, topic limits, output checks.
- Test adversarially before launch and after changes; see AI red teaming.
- Keep a person on the decisions that matter, and train them not to rubber-stamp.
- Watch the vendor: model versions, data use, change notices and exit options.
- Monitor in production for drift, complaints and incidents, and keep the records.
For Canadian banks and insurers, a generative model used in a business process is a model under OSFI's E-23, and OSFI's AI bulletin adds technology and cyber practices for generative and agentic AI. MINERVA keeps each generative AI system on the same record as every other model, with its rating, tests, findings and approvals. See AI governance software.
Related questions
What are the main risks of generative AI?
NIST AI 600-1 lists twelve, including confabulation (confident false output), data privacy, information security, harmful bias, human over-reliance, intellectual property and risks from third-party models and components.
What is NIST AI 600-1?
The Generative AI Profile of NIST's AI Risk Management Framework, published on 26 July 2024. It describes twelve risks unique to or worsened by generative AI and suggests actions to manage them.
What is confabulation in AI?
NIST's term for confidently stated but false or erroneous content, commonly called hallucination.
Is generative AI covered by OSFI E-23?
Yes. E-23's definition of a model includes AI and machine learning methods, so a generative model used in a business process is in scope from 1 May 2027.
How can a bank reduce generative AI risk?
Inventory every use, set a staff policy, ground and check outputs, put guardrails around inputs and outputs, red team before launch, keep people on consequential decisions, manage vendors closely and monitor in production.
Sources
- NIST — AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (26 July 2024)
- NIST — AI Risk Management Framework
- OSFI — Guideline E-23, Model Risk Management (2027)
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.