What is AI red teaming?
By Nick Jang, founder of MINERVA Studio ·
The short answer
AI red teaming is deliberately trying to make an AI system misbehave so its weaknesses are found before customers or attackers find them. NIST describes it as a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and with the system’s developers. For a chatbot or agent, that means trying to make it leak data, break its rules, give harmful or false answers, or take actions it shouldn’t. Every successful attack becomes a finding with an owner and a fix, and the test is run again after the fix.
How it differs from other testing
A benchmark or evaluation measures how well a model does on a fixed set of tasks. A penetration test looks for ways into the systems around it. Red teaming is adversarial and open-ended: testers think like a hostile or careless user and keep pushing where the system gives a little.
The term comes from military and security exercises, where a red team plays the opponent. With AI, it also covers failures nobody planned: a well-meaning customer who gets a confidently wrong answer about their mortgage is a red team finding too.
What testers try
| Target | What the tester tries |
|---|---|
| Prompt injection and jailbreaks | Getting the model to ignore its instructions, directly or through text hidden in a document or web page it reads. |
| Data leakage | Extracting the system prompt, other users’ data, internal documents or secrets. |
| Harmful or off-policy output | Advice the firm mustn’t give, offensive content, or promises the business can’t keep. |
| False answers | Pushing it to invent policies, fees, citations or facts, and checking whether it admits uncertainty. |
| Bias | Asking the same thing for different people and comparing the answers. |
| Unsafe actions | For agents: tricking it into calling a tool it shouldn’t, with arguments it shouldn’t, or exceeding its limits. |
| Abuse and cost | Huge or repeated requests that run up costs or knock the service over. |
The OWASP Top 10 for LLMs and the agentic top 10 make good starting lists.
How an exercise runs
OWASP’s GenAI Red Teaming Guide (January 2025) splits the work into four areas: the model itself, the implementation around it (prompts, guardrails, retrieval), the infrastructure, and the system’s behaviour at runtime. A typical exercise:
- Scope: which system, which version, what’s in and out of bounds, and what counts as a failure for this use.
- Threat model: who would attack it or misuse it, and what they’d want.
- Test: manual attacks by people with domain knowledge, plus automated tools that generate and mutate thousands of attempts.
- Record: every successful attack with the exact input, the output, the version and the severity.
- Fix and retest: the builders fix; the red team confirms the fix and checks it didn’t break something else.
Domain experts matter more than people expect. A security tester can find a jailbreak; it takes someone from the credit team to notice that an answer is wrong in a way that would mislead a borrower.
Tools
Several free, open-source tools automate the repetitive part: PyRIT from Microsoft, garak from NVIDIA and promptfoo generate attack prompts, run them against your system and score the responses. They speed things up but don’t replace people. Automated attacks find the known patterns; people find the failures specific to your business.
NIST’s Generative AI Profile (NIST AI 600-1) suggests red teaming as one way to measure resilience against prompt injection and other attacks, and it treats red teaming results as evidence for a risk management program.
After the exercise
Red teaming is only worth the findings that get fixed. Each one should land on the system’s record with a severity, an owner, a due date and the evidence, and a serious one should stop the system from going live until it’s closed. Run it again before major releases, when the underlying model changes and on a regular cycle for high-risk systems.
In a bank, red team results are part of the validation evidence OSFI’s E-23 expects for AI models. MINERVA keeps findings on each model, links them to the test that found them, and won’t let a model be approved while a high-severity finding is open. See AI governance software.
Related questions
What is the difference between AI red teaming and penetration testing?
A penetration test looks for ways to break into systems and networks. AI red teaming attacks the AI’s behaviour: making it leak data, break its rules, give harmful or false answers or take unsafe actions. Many exercises do both.
What tools are used for AI red teaming?
Open-source tools such as Microsoft’s PyRIT, NVIDIA’s garak and promptfoo automate attack generation and scoring. They are combined with manual testing by people who know the domain.
How often should AI systems be red teamed?
Before launch, before major releases, whenever the underlying model changes, and on a regular cycle for high-risk systems.
Who should do AI red teaming?
People independent of the builders, mixing security skills with domain experts who can recognise a harmful or misleading answer in context. Larger organizations have internal teams; others use specialist firms.
Is AI red teaming required by law?
Not generally in Canada. The EU AI Act requires adversarial testing for general-purpose AI models with systemic risk, and regulators such as OSFI expect AI models to be tested and validated before use.
Sources
- NIST CSRC Glossary — artificial intelligence red-teaming
- OWASP GenAI Security Project — GenAI Red Teaming Guide (January 2025)
- NIST — AI Risk Management Framework: Generative AI Profile (NIST AI 600-1)
- Microsoft — PyRIT
- NVIDIA — garak
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.