21.2.1 Trust in AI systems

Trust is not an abstract feeling; in the context of AI systems, it is an emergent property derived from verifiable technical attributes. Your role as a red teamer is to treat trust as a critical system component and subject it to rigorous, structured failure analysis. Where a developer sees a feature, you must see a potential vector for trust erosion. This chapter deconstructs trust into a taxonomy of vulnerabilities, providing a framework for systematically dismantling and, ultimately, hardening it.

A Red Teamer’s Taxonomy of Technical Trust

Instead of viewing trust holistically, we dissect it into four core, interdependent pillars. Each pillar represents a domain of technical properties that can be empirically tested and broken. A failure in one pillar often precipitates a collapse in others, leading to a total loss of system trustworthiness. The pillars are:

• Reliability: Does the system perform its function correctly and consistently?
• Explainability: Can the system’s reasoning be understood and verified?
• Robustness: Can the system withstand adversarial or unexpected inputs?
• Fairness: Does the system produce equitable outcomes?

Your objective is to identify and exploit vulnerabilities within each of these domains, demonstrating concrete scenarios where placing trust in the AI would be a critical error.

Pillar 1: Reliability and Predictability Failures

Reliability is the foundation. If an AI system cannot be relied upon to perform its core function under expected conditions, all other properties are moot. High-level accuracy metrics (e.g., 99% accuracy) often conceal dangerous reliability failures.

Vulnerability: Performance Cliffs

A performance cliff is a catastrophic failure in response to a minor, often imperceptible, change in input that is still well within the expected data distribution. These are not adversarial examples; they are brittle failures on legitimate data.

Your task is to hunt for these cliffs. This involves moving beyond random validation sets and generating inputs that probe the boundaries of learned concepts.

# Pseudocode: Probing for a performance cliff in an image classifier
model = load_model('vehicle_classifier.h5')
base_image = load_image('standard_sedan.jpg')
base_prediction, base_confidence = model.predict(base_image)

# Introduce a minor, non-adversarial perturbation (e.g., change lighting)
for brightness_delta in range(-10, 10):
    perturbed_image = adjust_brightness(base_image, brightness_delta)
    new_prediction, new_confidence = model.predict(perturbed_image)

    # A cliff is a sudden, drastic change in prediction or confidence
    if new_prediction != base_prediction and new_confidence < 0.2:
        print(f"Performance cliff detected at brightness delta: {brightness_delta}")
        print(f"Prediction changed from {base_prediction} to {new_prediction}")
        break

Pillar 2: Exploiting Flawed Explainability (XAI)

Explainability (XAI) methods are often presented as a solution for building trust. For a red teamer, they are a new attack surface. An explanation that is itself misleading is more dangerous than no explanation at all, as it fosters a false sense of security.

Vulnerability: Adversarial Explanations

The goal here is not to change the model’s output, but to corrupt its explanation. You can craft an input that the model classifies correctly, but for which the XAI technique (like LIME or SHAP) highlights completely irrelevant features as being important. This proves the model’s reasoning is not aligned with human intuition, even when its answer is correct.

Pillar 3: Robustness and Security Breakdowns

This pillar covers the more traditional domain of adversarial machine learning. Trust is impossible if the system is fragile and easily manipulated by a malicious actor. Your goal is to demonstrate that this fragility is not a theoretical concern but a practical, exploitable vulnerability.

Table 1: Taxonomy of Robustness Attacks on Trust

Attack Type	Red Team Objective	Type of Trust Violated
Evasion Attack	Craft a malicious input (e.g., adversarial example) to cause a specific misclassification at inference time.	Output Integrity: The system’s predictions cannot be trusted in an adversarial environment.
Model Extraction	Query the model API to reconstruct a functional copy of the proprietary model.	Intellectual Property: The organization’s investment and model secrecy are compromised.
Model Inversion	Use model outputs to reconstruct sensitive information about the private data it was trained on.	Data Confidentiality: The system cannot be trusted to protect the privacy of its training data.
Data Poisoning	Inject malicious samples into the training data to create a backdoor that can be triggered later.	Behavioral Integrity: The model’s fundamental logic has been corrupted and cannot be trusted.

Pillar 4: Amplifying Unfairness and Bias

An AI system that is reliable, explainable, and robust for one demographic but not another is fundamentally untrustworthy. Fairness is not a “nice-to-have” ethical feature; it is a core component of system reliability. Your objective is to find and weaponize algorithmic bias to demonstrate discriminatory harm.

Vulnerability: Subgroup Reliability Collapse

This occurs when a model performs well on aggregate metrics but fails spectacularly for a specific, often underrepresented, subgroup. This is a common and high-impact vulnerability.

The red teaming exercise involves identifying sensitive attributes (e.g., race, gender, age) and then systematically evaluating model performance for each subgroup. The goal is to find a subgroup where the error rate is unacceptably high, proving the system is not equitable.

# Pseudocode: Auditing for subgroup reliability collapse
dataset = load_validation_data_with_demographics()
model = load_model('loan_approval_model.h5')

subgroups = dataset.get_unique_values('ethnicity')
results = {}

for group in subgroups:
    group_data = dataset.filter(ethnicity=group)
    predictions = model.predict(group_data.features)
    error_rate = calculate_error(predictions, group_data.labels)
    results[group] = error_rate
    print(f"Error rate for {group}: {error_rate:.2%}")

# Identify significant disparities that indicate a fairness/reliability failure
max_error = max(results.values())
min_error = min(results.values())
if max_error / min_error > 2.0:
    print("n[!] CRITICAL: Subgroup reliability collapse detected!")

Synthesis: The Trust Cascade Failure

The most devastating attacks demonstrate a cascade failure, where a vulnerability in one pillar triggers a collapse in the others. This paints a vivid picture of systemic untrustworthiness.

Consider a hiring model that learns a spurious correlation between a candidate’s resume format and their qualifications (an explainability failure). An attacker could then design resumes with this format to bypass the filter (a robustness failure). If this format is more common among applicants from a specific demographic, the system now systematically favors them (a fairness failure), leading to unreliable and biased hiring recommendations (a reliability failure).

By framing your findings in this narrative, you move from reporting isolated bugs to demonstrating a fundamental breakdown in the system’s capacity to be trusted. This is the ultimate goal of red teaming in the socio-technical context: to provide the empirical evidence needed to justify a verdict of “untrustworthy.”