While previous chapters focused on the “how” of breaking AI systems, this section pivots to the “why” and “what” dictated by emerging law. The European Union’s AI Act is a landmark piece of legislation that moves AI security from a “best practice” to a legal mandate for many systems. For a red teamer, this regulation is not just a compliance checklist; it’s a detailed blueprint of potential failure points and mandated defenses that you will be hired to test.
The Risk-Based Approach: Your New Triage Framework
The EU AI Act doesn’t treat all AI the same. It categorizes systems into a pyramid of risk, which directly dictates the level of scrutiny and, consequently, the scope of your red teaming engagements. Understanding this hierarchy is the first step to aligning your testing strategy with legal requirements.
Your work will almost exclusively focus on the High-Risk category. These are systems whose failure could have severe consequences for people’s health, safety, or fundamental rights. Think AI used in medical diagnostics, critical infrastructure management, law enforcement, or hiring.
Deep Dive: High-Risk AI Systems as a Red Team Roadmap
The AI Act imposes a set of mandatory requirements on high-risk systems before they can enter the EU market. For you, each requirement is a domain for adversarial testing. Your goal is to validate—or invalidate—the provider’s claims of compliance through rigorous, practical testing.
| Requirement (Article from AI Act) | Description | Your Red Teaming Mandate |
|---|---|---|
| Risk Management System (Art. 9) | Providers must establish, implement, and maintain a continuous risk management system throughout the AI’s lifecycle. | Review their risk assessments. Can you devise threats they missed? Test the mitigations they put in place for known risks. |
| Data and Data Governance (Art. 10) | Training, validation, and testing data must be relevant, representative, free of errors, and complete. Biases must be examined and mitigated. | Conduct data poisoning attacks. Perform algorithmic bias audits. Test for performance degradation on underrepresented subgroups. |
| Technical Documentation (Art. 11) | Detailed documentation must be created before the system is placed on the market, proving compliance with all requirements. | Use the documentation as a guide. Does the system behave as documented? Find the gaps between documentation and reality. This is a prime area for model inversion or extraction attacks. |
| Record-Keeping (Logging) (Art. 12) | Systems must have capabilities to automatically record events (logs) to ensure traceability of the system’s functioning. | Test the integrity of the logs. Can they be tampered with? Do they capture enough information for forensic analysis after an incident you create? Can you trigger an incident without leaving a proper trace? |
| Transparency and Provision of Information (Art. 13) | The system must be designed to be sufficiently transparent to enable users to interpret the system’s output and use it appropriately. | Generate ambiguous inputs and analyze the explanations. Are they coherent? Could a user be misled by the provided information into making a poor decision? Test the system’s explainability (XAI) methods for fragility. |
| Human Oversight (Art. 14) | Systems must be designed to be effectively overseen by humans, including the ability to intervene or stop the system. | Design scenarios to test the “human-in-the-loop.” Can you create outputs that trick the human overseer? Can you cause the system to fail so quickly that human intervention is impossible? Test the “stop” button under duress. |
| Accuracy, Robustness, and Cybersecurity (Art. 15) | Systems must achieve an appropriate level of accuracy, be resilient against errors, and be secure against attempts to alter their use or performance. | This is classic adversarial ML territory. Perform evasion attacks (e.g., adversarial patches), model extraction, and fuzzing. Test performance against noisy or out-of-distribution data. Probe for traditional cybersecurity vulnerabilities in the supporting infrastructure. |
The Conformity Assessment: Your Test Report’s Destination
Before a high-risk AI system can be sold, it must undergo a “conformity assessment.” In many cases, this is a self-assessment by the provider, but for certain critical applications, a third-party audit (a “Notified Body”) is required. Your red team report is a critical piece of evidence in this process. It provides the objective, adversarial validation that goes beyond theoretical checks.
Imagine your findings being used to answer a conformity checklist. Your work directly translates into demonstrating due diligence or, conversely, flagging a system as non-compliant.
// Pseudocode for a simplified conformity check
function check_compliance(ai_system, red_team_report) {
// Requirement: Robustness against adversarial inputs (Art. 15)
let robustness_test = red_team_report.find_test("Evasion Attack Resilience");
if (robustness_test.success_rate < 0.95) {
return { compliant: false, reason: "Failed robustness check under adversarial conditions." };
}
// Requirement: Transparency / Explainability (Art. 13)
let explainability_test = red_team_report.find_test("Explanation Coherency");
if (!explainability_test.is_coherent) {
return { compliant: false, reason: "Generated explanations are misleading for edge cases." };
}
// Requirement: Human Oversight effectiveness (Art. 14)
let oversight_test = red_team_report.find_test("Human Override Latency");
if (oversight_test.latency_ms > 500) {
return { compliant: false, reason: "Human intervention is too slow in simulated critical failures." };
}
// If all checks pass...
return { compliant: true, reason: "All critical red team checks passed." };
}
Your role is to generate the empirical data—the `red_team_report` in this example—that makes this assessment meaningful. Without adversarial testing, a conformity assessment is just a paper exercise. You provide the ground truth.