Beyond the general terms of a service contract, the Service Level Agreement (SLA) provides the technical and operational backbone of your engagement. It translates abstract promises of “performance” and “security” into measurable, enforceable metrics. For AI systems, a standard IT SLA is dangerously insufficient; it fails to capture the unique failure modes and performance characteristics of machine learning models.
Core Concept: An AI-centric SLA moves beyond uptime and response time. It must quantify the model’s functional correctness, its resilience to adversarial pressure, and the speed at which security mechanisms can detect and respond to AI-specific threats.
From Infrastructure to Intelligence: Redefining SLA Metrics
Traditional SLAs focus on the health of the infrastructure hosting an application. While important, this perspective misses the point with AI. An API endpoint can return a `200 OK` status with perfect uptime while the model behind it is producing nonsensical, biased, or malicious output. Your SLA must account for the intelligence layer.
As a red teamer, the SLA is not just a legal document; it’s a rulebook for your engagement. Your objective is often to push the system until it violates one of these agreed-upon thresholds. The SLA defines what “broken” means.
Key Categories for AI System SLAs
A robust SLA for an AI service should be structured around metrics that cover both the service delivery and the model’s core function.
1. Service Availability and Performance
This is the most traditional category, but with critical AI-specific nuances.
- API Uptime: The percentage of time the model’s endpoint is available and responsive. This is standard.
- Inference Latency (Baseline): The time taken to process a single, standard request under normal load conditions. This sets the performance expectation.
- Inference Latency (Under Load): The maximum acceptable latency when the system is under a specified level of stress (e.g., queries per second). This is a key target for resource exhaustion and denial-of-service tests.
- Error Rate: The percentage of requests that result in a server-side error (5xx codes), distinguishing infrastructure failures from model execution failures.
2. Model Quality and Integrity
This is where AI SLAs diverge significantly from the norm. These metrics measure the “correctness” of the model’s output.
- Accuracy/Performance Drift: A commitment that the model’s accuracy (or other relevant metric like F1 score, precision) on a golden validation dataset will not degrade by more than a specified percentage (e.g., 2%) over the contract period.
- Output Format Adherence: For models generating structured data (e.g., JSON), this metric defines the percentage of responses that must conform to the specified schema. This is a target for prompt injection attacks that aim to break output structure.
- Content Policy Adherence: The percentage of outputs that must comply with safety and content policies (e.g., no hate speech, no PII leakage). This is directly tested during jailbreaking and privacy attacks.
3. Security and Incident Response
This section defines how the service provider must react to security events, including those simulated by your red team.
- Time to Detect (TTD): The maximum time allowed between the initiation of a known adversarial attack pattern and its detection by the provider’s monitoring systems.
- Time to Respond (TTR): The time from detection to the initiation of a mitigation action.
- Reporting Cadence: The frequency of updates required during a simulated high-severity incident.
Comparing Traditional and AI-Centric SLA Metrics
The following table highlights the conceptual shift required when drafting or evaluating an SLA for an AI system.
| Metric Category | Traditional IT SLA Example | AI Red Teaming SLA Example | Why It’s Different |
|---|---|---|---|
| Availability | 99.95% server uptime. | 99.9% API availability AND <0.5% critical model execution error rate. | Focuses on functional correctness, not just network reachability. The model can fail even if the server is up. |
| Performance | <250ms average API response time. | Baseline inference latency <150ms; 95th percentile latency <500ms under simulated adversarial load (defined in scope). | Defines performance under adversarial conditions, not just benign traffic. |
| Security | Time to acknowledge critical security alert: 1 hour. | Time to detect known prompt injection pattern: <10 minutes. | Measures the security system’s specific capability to identify and react to AI-native threats. |
| Quality | Not applicable. Assumes application logic is static. | Model accuracy on reference dataset must not drift more than 2% month-over-month. | Directly measures the core function of the AI, acknowledging that its performance can degrade over time. |
Example SLA Clause
A well-defined SLA clause is specific, measurable, and includes a remedy for non-compliance. Here is an example of a clause addressing model quality under adversarial pressure.
5.1 Content Policy Compliance: The `generative-chat-v4` model service must filter responses against the “Harmful Content Policy” (Appendix C). The rate of policy-violating responses, when tested against the “Safety Benchmark Prompt Set” (Appendix D), shall not exceed 0.1%.
5.2 Measurement: Compliance will be measured via a random sampling of 10,000 prompts from the benchmark set, conducted by the client on a monthly basis.
5.3 Remedy for Non-Compliance: Should the rate of policy-violating responses exceed 0.1% in a given month, a service credit of 5% of that month’s service fee will be applied to the client’s account. If the rate exceeds 0.5%, the provider is required to conduct a Root Cause Analysis (RCA) and deliver a remediation plan within 10 business days.
For a red teamer, this clause is a clear objective. Your goal is to develop techniques (e.g., jailbreaking, complex role-playing prompts) that cause the model to violate the 0.1% threshold, thereby triggering the SLA’s remedy clause and demonstrating a concrete business impact.