This appendix provides a quick-reference glossary of common acronyms used throughout the fields of AI security, adversarial machine learning, and general cybersecurity. Familiarity with this terminology is essential for effective communication and a deeper understanding of red teaming concepts.
Acronym Glossary
The following table lists acronyms in alphabetical order for easy reference. The “Relevance in AI Red Teaming” column provides specific context for how each term applies to the assessment of AI systems.
| Acronym | Full Name | Relevance in AI Red Teaming |
|---|---|---|
| AGI | Artificial General Intelligence | A hypothetical form of AI with human-like cognitive abilities. Red teaming AGI concepts involves exploring long-term, existential risks and complex, multi-system failure modes. |
| API | Application Programming Interface | The primary method for interacting with deployed AI models. Red teaming focuses on exploiting API vulnerabilities such as insecure direct object references, rate limiting bypasses, and business logic flaws. |
| ASR | Adversarial Success Rate | A key metric used to quantify the effectiveness of an adversarial attack, representing the percentage of adversarial inputs that successfully deceive the target model. |
| AST | Application Security Testing | The overall practice of finding security vulnerabilities in software. For AI, this encompasses SAST, DAST, and IAST applied to the MLOps pipeline, model-serving infrastructure, and related code. |
| C2 / C&C | Command and Control | Infrastructure used by attackers to manage compromised systems. In AI attacks, a malicious model could be designed to exfiltrate sensitive data or receive new instructions from a C2 server. |
| CNN | Convolutional Neural Network | A class of deep neural networks, highly effective for analyzing visual imagery. They are a common target for adversarial image-based attacks like patch attacks and evasion techniques. |
| CVE | Common Vulnerabilities and Exposures | A standardized list of publicly disclosed security vulnerabilities. Red teamers check the AI system’s entire stack (e.g., PyTorch, TensorFlow, Docker, Kubernetes) for relevant CVEs. |
| CVSS | Common Vulnerability Scoring System | An open standard for assessing the severity of security vulnerabilities. It helps red teams prioritize findings based on potential impact. |
| DAST | Dynamic Application Security Testing | A “black-box” testing methodology where an application is tested in its running state. It’s used to find vulnerabilities in the web interfaces or APIs that expose AI models. |
| FGSM | Fast Gradient Sign Method | A foundational “white-box” adversarial attack designed to generate adversarial examples. It works by adding a small perturbation in the direction of the loss function’s gradient. |
| GAN | Generative Adversarial Network | A class of machine learning models used for generative tasks. Can be used offensively to create realistic deepfakes for social engineering or defensively to generate synthetic training data. |
| IAM | Identity and Access Management | The security discipline that ensures the right individuals have the right access to resources. Misconfigured IAM policies are a primary target for escalating privileges within the cloud environment hosting an AI model. |
| LLM | Large Language Model | An AI model trained on vast text datasets to understand and generate language. As the core of modern generative AI, LLMs are the primary target for prompt injection, jailbreaking, and data extraction attacks. |
| MLOps | Machine Learning Operations | A set of practices for deploying and maintaining machine learning models in production reliably and efficiently. The entire MLOps pipeline, from data ingestion to model deployment, is a critical attack surface. |
| OWASP | Open Web Application Security Project | A non-profit foundation focused on improving software security. The OWASP Top 10 for LLM Applications is an essential framework for structuring AI red teaming engagements. |
| PGD | Projected Gradient Descent | A powerful, iterative “white-box” adversarial attack. It’s a stronger version of FGSM and is often used as a robust benchmark to evaluate a model’s security posture against adversarial examples. |
| PII | Personally Identifiable Information | Data that can be used to identify a specific individual. A key goal for red teamers is to test whether an AI model can be manipulated into leaking PII from its training data (membership inference or reconstruction attacks). |
| RLHF | Reinforcement Learning from Human Feedback | A training technique used to align LLMs with human values and safety constraints. Red team engagements are often designed to find clever inputs that bypass the safety guardrails implemented via RLHF. |
| SAST | Static Application Security Testing | A “white-box” testing methodology where an application’s source code is analyzed for security flaws without executing it. Useful for finding vulnerabilities in the code that implements or serves the AI model. |
| TTP | Tactics, Techniques, and Procedures | A framework for describing and analyzing attacker behavior. AI red teamers emulate the TTPs of known threat actors to provide a realistic assessment of an organization’s defenses. |