Navigating the European Union’s AI Act requires a systematic approach to ensure your systems are developed and deployed responsibly. This checklist provides a structured framework for assessing your AI system’s alignment with the Act’s core requirements, specifically tailored for a red team and security perspective.
Use this tool to identify potential gaps in compliance, guide your testing strategy, and document due diligence. It translates legal obligations into actionable verification steps.
Disclaimer: This checklist is an educational and operational tool, not a substitute for legal advice. The AI Act is a complex and evolving regulation. Always consult with qualified legal counsel for formal compliance guidance.
Step 1: Risk Classification
Your obligations under the AI Act are determined by the risk category of your AI system. Before proceeding, you must accurately classify your system. The Act defines four primary levels:
- Unacceptable Risk: Systems that are explicitly banned (e.g., social scoring by public authorities, real-time remote biometric identification in publicly accessible spaces for law enforcement, with narrow exceptions). Compliance Check: Confirm your system does not fall into any banned categories listed in Article 5.
- High-Risk: Systems used in critical sectors or for crucial purposes listed in Annex III (e.g., medical devices, critical infrastructure management, recruitment, credit scoring) or systems that are a safety component of a product covered by specific EU legislation (Annex II). These systems face the most stringent requirements.
- Limited Risk: Systems that require specific transparency obligations, such as chatbots, emotion recognition systems, and systems that generate “deep fakes.” Users must be aware they are interacting with an AI.
- Minimal/No Risk: All other AI systems (e.g., spam filters, AI in video games). Most AI systems are expected to fall into this category, with no specific legal obligations, though voluntary codes of conduct are encouraged.
Step 2: Checklist for High-Risk AI Systems
If your system is classified as high-risk, it must meet a comprehensive set of mandatory requirements before being placed on the market. This table breaks down those core obligations and suggests how a red team can scrutinize them.
| Requirement Area | Key Provision (Article) | Compliance Check / Key Question | Red Team Scrutiny Point |
|---|---|---|---|
| Risk Management System | Article 9 | Is there a continuous, iterative risk management system established, implemented, and maintained throughout the AI system’s lifecycle? Does it identify, estimate, and evaluate known and foreseeable risks? | Design scenarios to trigger unforeseen risks. Test for edge cases and emergent behaviors that the documented risk management plan fails to cover. Can you introduce a novel threat that bypasses existing mitigation measures? |
| Data and Data Governance | Article 10 | Are the training, validation, and testing datasets relevant, representative, free of errors, and complete? Is there appropriate governance to examine for and mitigate potential biases? | Attempt data poisoning attacks to introduce biases or backdoors. Probe the system with underrepresented subgroup data to expose performance disparities and discriminatory outcomes. Test data anonymization techniques for re-identification vulnerabilities. |
| Technical Documentation | Article 11 & Annex IV | Is comprehensive technical documentation maintained and available to authorities? Does it describe the system’s purpose, capabilities, limitations, and the methodologies used for design and validation? | Review documentation for inconsistencies, omissions, or unsubstantiated claims about performance and safety. Can you demonstrate a system behavior that contradicts the official documentation? |
| Record-Keeping (Logging) | Article 12 | Does the system have the capability to automatically record events (“logs”) throughout its operation to ensure traceability of its functioning? | Attempt to tamper with or disable logging mechanisms. Execute actions designed to be “invisible” to the logging system. Assess if logs are sufficient to reconstruct a security incident or a harmful decision-making process. |
| Transparency and Provision of Information | Article 13 | Are users provided with clear and adequate instructions for use? Does the information specify the system’s intended purpose, level of accuracy, and the nature of human oversight required? | Conduct user-centric tests to determine if the provided information is truly understandable and actionable. Can a non-expert user misinterpret the instructions, leading to unsafe operation? Is the system’s confidence score or output easily misconstrued? |
| Human Oversight | Article 14 | Are the measures for human oversight appropriate for the risks? Can a human effectively intervene, override, or stop the system, especially in case of unexpected behavior? | Create scenarios that test the limits of human oversight. Can you induce a system failure so rapidly that human intervention is practically impossible? Test “automation bias” by presenting misleading AI recommendations to see if human overseers comply. |
| Accuracy, Robustness, and Cybersecurity | Article 15 | Does the system perform at an appropriate level of accuracy for its intended purpose? Is it resilient against errors, faults, or inconsistencies? Is it secure against attempts to alter its use or performance? | Launch a full spectrum of adversarial attacks: evasion (adversarial examples), model inversion, membership inference, and model stealing. Perform penetration testing on the supporting infrastructure (APIs, databases). Test robustness against corrupted or out-of-distribution inputs. |
Step 3: Checklist for Limited Risk & GPAI Systems
Even if your system is not high-risk, other obligations may apply.
Transparency Obligations (Limited Risk)
| System Type | Key Provision (Article) | Compliance Check / Key Question | Red Team Scrutiny Point |
|---|---|---|---|
| Systems interacting with humans (e.g., chatbots) | Article 52(1) | Are users clearly and timely informed that they are interacting with an AI system, unless it is obvious from the context? | Attempt to design interactions where the AI disclosure is obscured, delayed, or ambiguous. Can you make the chatbot convincingly “pass” as human for a duration that could be deceptive? |
| Deep Fakes (AI-generated content) | Article 52(3) | Is AI-generated audio, image, video, or text content that falsely appears to be authentic or truthful (“deep fakes”) labeled as artificially generated or manipulated? | Test the watermarking or labeling mechanism. Can it be easily removed or altered? How does the system handle content that is subtly manipulated versus overtly fabricated? |
General-Purpose AI (GPAI) Models
Providers of GPAI models, especially those deemed to have systemic risk, have their own set of obligations.
- Technical Documentation: Is documentation available for downstream providers, including the model’s training process and testing results?
- Copyright Policy: Is there a policy in place to respect EU copyright law, particularly regarding the data used for training?
- Systemic Risk Obligations (for powerful models): This includes performing model evaluation, assessing and mitigating systemic risks, tracking incidents, and ensuring adequate cybersecurity. Red teams should focus heavily on adversarial testing, evaluating mitigations for dangerous capabilities, and testing incident response protocols.