AI security and testing
AI red team: what it is, how a test round works and who should do it
A plain guide for security, risk and product leads planning to test an AI system before customers do. Every fact about red teaming comes from Microsoft's or OWASP's own pages as read on 8 November 2026.
An AI red team is a group that tests an AI system by trying to make it fail: produce harmful output, leak data or take an action it should not. Microsoft's guidance, read 8 November 2026, calls red teaming a best practice but not a replacement for systematic measurement. Plan who tests, what they test and how findings are recorded.
plain answer
What is an AI red team?
An AI red team tests an AI system by deliberately trying to make it fail. Microsoft's guidance says the term red teaming has historically described systematic adversarial attacks for testing security vulnerabilities, and that with large language models it has extended to describe many kinds of probing, testing and attacking of AI systems. It adds that both ordinary and adversarial use can produce harmful output, so a red team includes people who attack the system and people who use it the way a customer would. OWASP's Gen AI Security Project describes its AI red teaming work as testing generative AI systems through adversarial methods, and lists a Red Teaming Manual among its guides. The aim is to find harms and understand the risk surface.
Sources: Microsoft Learn, Planning red teaming for large language models (LLMs) and their applications; OWASP Gen AI Security Project, AI Red Teaming initiative (read 8 November 2026).
plain answer
How does an AI red team test work?
Microsoft's guidance splits a red team exercise into planning, testing and reporting. Before testing, decide who will test, what they will test and how findings will be recorded. The testing can target the base model, your own application, or both, and the guidance advises testing the production interface as much as possible because it most closely resembles real use. It suggests starting with open-ended testing to uncover a wide range of harms, writing a list of the harms found, then running guided rounds against that list and repeating. After each round, share a short report with the top issues and a link to the raw data. Treat the examples as findings, not as a measure of how common a harm is.
Source: Microsoft Learn, Planning red teaming for large language models (LLMs) and their applications (read 8 November 2026).
plain answer
Who should be on an AI red team?
Microsoft recommends a diverse group with experience across disciplines, for example experts in AI, social sciences and security, matched to the product's domain. Its example is a chatbot for health care providers, where medical experts can help identify risks in that field. It also recommends mixing two mindsets: testers with an adversarial mindset and security-testing experience, who are essential for understanding security risks, and ordinary users who were not involved in building the application, who can see harms regular users would meet. Testers can be assigned to specific harms or to specific features, and it helps to give them clear written instructions and one shared place to record the input, the output and the date for each finding.
Source: Microsoft Learn, Planning red teaming for large language models (LLMs) and their applications (read 8 November 2026).
plain answer
Is an AI red team enough on its own?
No. Microsoft states that red teaming is not a replacement for systematic measurement and mitigation work. Its recommended order is an initial round of manual red teaming first, to identify harms and understand the risk surface, then systematic measurement to check that the fixes work. For AI agents there is a second question: what the agent is allowed to do. OWASP describes excessive agency as the vulnerability that lets damaging actions happen in response to unexpected, ambiguous or manipulated model output, and names three root causes: excessive functionality, excessive permissions and excessive autonomy. We recommend reducing those before any test, so the test checks whether the limits hold.
Sources: Microsoft Learn, Planning red teaming for large language models (LLMs) and their applications; OWASP Gen AI Security Project, LLM06:2025 Excessive Agency (read 8 November 2026).
kriv role
What does Kriv AI do about AI red teaming?
We run red-team passes as part of validating the AI systems we build or assess. We are not a penetration-testing firm, and we agree the scope of any testing with you before it starts. Work is scoped and priced after a short discovery call.
Sources
Cited sources
Related resources
Continue exploring
Straight answers
Frequently asked questions about AI red team: what it is, how a test round works and who should do it
What is an AI red team?
It is a group that tests an AI system by trying to make it fail, for example by producing harmful output or taking an action it should not. Microsoft describes the practice as a best practice in developing systems that use large language models (read 8 November 2026).
Is AI red teaming the same as penetration testing?
Not exactly. Microsoft says red teaming historically meant systematic adversarial attacks to test security vulnerabilities and has widened with large language models to cover many kinds of probing, testing and attacking of AI systems, including harmful content from ordinary use (read 8 November 2026).
Who should do AI red teaming?
Microsoft recommends a diverse group with expertise across disciplines, mixing testers with an adversarial mindset and security experience with ordinary users who did not build the application (read 8 November 2026).
When should you red team an AI system?
Microsoft advises an initial round of manual red teaming before systematic measurement, then repeated testing with and without mitigations in place, and testing on the production interface as much as possible (read 8 November 2026).
Does a red team find every problem?
No. Microsoft says the role of red teaming is to expose and raise understanding of the risk surface, and that specific examples should not be read as a metric for how widespread a harm is (read 8 November 2026).
Talk to the team that would do the work
Bring your requirements to a working session with the person who'll actually deliver.
Book a Discovery Call