Until now, the first line of defense in artificial intelligence security has been human expertise and the creativity of ‘red teams’. However, researchers at the Oak Ridge National Laboratory’s (ORNL) Center for Artificial Intelligence Security Research (CAISER) have developed a tool that could fundamentally change this paradigm. The framework, named Photon, uses exascale computing power to automatically uncover hidden vulnerabilities in AI models on an industrial scale.
Running on ORNL’s Frontier supercomputer, the system can execute 60,000 ‘jailbreak’ prompts per hour, far exceeding the performance of any human team. This technological leap not only increases the speed of testing but also the depth and complexity of the flaws that can be discovered.
Do you have a question about AI security? You can reach us here:
How Does Photon Work? The Art of Attack Optimization
Photon is based on a previous ORNL technology called DeepHyper. Originally, DeepHyper was designed to aid in the training of large neural networks by finding optimal network parameters. The CAISER researchers ingeniously repurposed this concept: instead of optimizing the model, they perfect the attack.
The process begins by applying publicly known attack vectors from scientific literature against a target model. Photon then uses the DeepHyper framework to automatically, asynchronously, and in a decentralized manner, search for the hyperparameters that result in the most effective attack. As Edmon Begoli, director of ORNL’s CAISER, put it:
“It might sound devious, but it’s worked very well.”
The goal is not just to find a vulnerability, but to identify the fastest and most efficient methods for a ‘jailbreak’. According to Jack Hutchins, an ORNL robust AI engineer: “Since our goal is to find highly effective jailbreaks, finding the parameters that have the most effect quickly speeds up our search for effective jailbreaks.”
The Significance of Exascale
Photon’s power lies not only in its algorithm but also in the raw force behind it. Frontier, ORNL’s exascale supercomputer, provides the computational capacity to map the attack parameter space at lightning speed. The system can maintain over 95 percent resource utilization even when running on 1,920 GPUs.
This figure alone is an impressive engineering achievement. Jack Hutchins highlights the challenge: “When we’re talking about running something at this scale, it becomes difficult to use as much of the available compute power as possible. Since you are running at such a large scale, eliminating resource downtime is not trivial.” The 95% utilization rate means the system is working almost continuously, minimizing idle time, which is crucial for efficiency.
AIQ Analysis: What This Means for Corporate AI Security
From an AIQ standpoint, the emergence of Photon is a clear signal to the market: the methodology of AI security auditing and red teaming is about to undergo a fundamental change. The era of manual, ad-hoc testing is slowly coming to an end, to be replaced by automated, scaled, and continuous vulnerability discovery.
OWASP LLM Top 10 Context
In a corporate context, this means that systems like Photon will enable systematic testing for a significant portion of the OWASP LLM Top 10 vulnerabilities. Specifically:
- LLM01: Prompt Injection: Photon’s 60,000 attempts per hour are precisely aimed at finding the parameters and prompt variations that can bypass a model’s safety filters.
- LLM04: Model Denial of Service: The system can identify complex, resource-intensive queries that could overload or paralyze a model.
- LLM06: Sensitive Information Disclosure: Scaled testing can efficiently uncover prompts that cause the model to leak sensitive personal or business data from its training set.
EU AI Act and GDPR Compliance
From AIQ’s perspective, technologies like Photon will soon become part of the ‘state of the art’, which will have a direct impact on regulatory compliance. The EU AI Act requires high-risk AI systems to be robust and secure. In the future, it will be difficult for a company to defend its due diligence during an audit if it relied solely on manual testing by a small red team, while industrial-scale, automated solutions are available on the market. This technology raises the bar for mandatory risk assessments and compliance procedures.
The situation is similar from a GDPR perspective. Before deploying an AI model, a Data Protection Impact Assessment (DPIA) requires a company to demonstrate that it has taken all reasonable technical and organizational measures to protect personal data. A tool that can systematically discover vulnerabilities leading to data leaks could become an essential auditing tool.
The Future: Human-Machine Collaboration
According to Edmon Begoli, Photon represents a paradigm shift in how we approach AI security. “By running coordinated, high-scale experiments, we can uncover hidden vulnerabilities far more efficiently than ever before.”
It is important to emphasize that such systems do not replace but rather augment human red team experts. While Photon handles parameter tuning and brute-force searching, human creativity will remain crucial for developing new, abstract attack logics and strategies. The future lies in a hybrid approach, where human intuition is supported by exascale computing power, ensuring that AI innovations do not also introduce unacceptable security risks.