An AI red teaming operation without a monitoring dashboard is like flying blind. While individual tools and scripts generate raw output, a centralized dashboard transforms that data into actionable intelligence. It provides the real-time situational awareness needed to track campaign progress, identify effective attack vectors, and understand the target model’s defensive posture. This section provides foundational code for building such a dashboard.
High-Level Architecture
Before diving into code, it’s essential to understand the data flow. A typical monitoring setup involves several components working in concert. Your testing tools generate events, which are captured, stored, and then served via an API to a frontend interface for visualization. This decouples the data collection from the presentation, allowing for a flexible and scalable system.
Backend Implementation: Data Endpoint
The backend’s primary role is to expose an API endpoint that provides aggregated data from your data store (e.g., a database like SQLite, PostgreSQL, or even a simple file). This example uses Python with the Flask micro-framework to create a simple endpoint serving mock data. In a real application, the `monitoring_data` dictionary would be populated by querying your database.
# file: dashboard_api.py
from flask import Flask, jsonify
from flask_cors import CORS # Needed for cross-origin requests from frontend
app = Flask(__name__)
CORS(app) # Allow requests from your frontend's domain
@app.route('/api/v1/metrics')
def get_metrics():
# In a real system, this data comes from a database.
# It would be updated by your API wrappers or webhook handlers.
live_data = {
"total_prompts": 15782,
"successful_attacks": 945,
"model_refusals": 4320,
"average_latency_ms": 2340,
"attack_breakdown": {
"prompt_injection": 450,
"data_extraction": 120,
"jailbreak": 375
}
}
return jsonify(live_data)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5001)To run this backend, you would install Flask (`pip install Flask Flask-Cors`) and execute the script (`python dashboard_api.py`). It will start a web server on port 5001, ready to serve data to your frontend.
Frontend Implementation: Visualization
The frontend consumes the data from the backend API and renders it in a human-readable format. This example uses plain HTML and JavaScript with the popular Chart.js library for creating visualizations. You can embed this code into a single HTML file.
This code creates a simple dashboard layout with key performance indicators (KPIs) and a bar chart visualizing the breakdown of successful attacks by type.
<!DOCTYPE html>
<html>
<head>
<title>AI Red Team Dashboard</title>
<!-- Include Chart.js from a CDN -->
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<!-- Link to the dashboard styles from this chapter's header -->
</head>
<body>
<div class="dashboard-container">
<h3>Live Operation Metrics</h3>
<div class="metric-grid">
<div class="metric-card">
<div id="total-prompts" class="metric-value">0</div>
<div class="metric-label">Total Prompts</div>
</div>
<div class="metric-card">
<div id="successful-attacks" class="metric-value">0</div>
<div class="metric-label">Successful Attacks</div>
</div>
<div class="metric-card">
<div id="model-refusals" class="metric-value">0</div>
<div class="metric-label">Model Refusals</div>
</div>
</div>
<div class="chart-container">
<canvas id="attackBreakdownChart"></canvas>
</div>
</div>
<script>
const API_URL = 'http://127.0.0.1:5001/api/v1/metrics';
let attackChart;
async function updateDashboard() {
try {
const response = await fetch(API_URL);
const data = await response.json();
// Update KPI Cards
document.getElementById('total-prompts').innerText = data.total_prompts.toLocaleString();
document.getElementById('successful-attacks').innerText = data.successful_attacks.toLocaleString();
document.getElementById('model-refusals').innerText = data.model_refusals.toLocaleString();
// Update Chart
const chartLabels = Object.keys(data.attack_breakdown);
const chartData = Object.values(data.attack_breakdown);
if (attackChart) {
attackChart.data.labels = chartLabels;
attackChart.data.datasets[0].data = chartData;
attackChart.update();
} else {
const ctx = document.getElementById('attackBreakdownChart').getContext('2d');
attackChart = new Chart(ctx, {
type: 'bar',
data: {
labels: chartLabels,
datasets: [{
label: 'Successful Attacks by Type',
data: chartData,
backgroundColor: ['#5bc0de', '#f0ad4e', '#d9534f'],
}]
},
options: { maintainAspectRatio: false }
});
}
} catch (error) {
console.error('Failed to fetch dashboard data:', error);
}
}
// Initial load and set to refresh every 5 seconds
updateDashboard();
setInterval(updateDashboard, 5000);
</script>
</body>
</html>Key Considerations
- Customization is Key: The metrics shown here are generic. Tailor your dashboard to the specific goals of your engagement. Are you testing for PII leakage? Add a metric for that. Are you focused on a specific vulnerability? Make its success rate a primary KPI.
- Real-time vs. Aggregated: The example polls the API every five seconds. For true real-time updates, consider using WebSockets to push data from the server to the client as events occur.
- Security: If this dashboard is exposed on a network, ensure your API endpoint is authenticated. You don’t want operational data to be publicly accessible.
- Scalability: For large-scale operations generating thousands of events per minute, a simple file or in-memory dictionary won’t suffice. You will need a robust database (like TimescaleDB or InfluxDB for time-series data) and potentially a more performant backend framework.