In a landmark case, the Brazilian Bar Association in Pará (OAB-PA) has precautionarily suspended two lawyers for 30 days. They are under investigation for allegedly attempting to manipulate an artificial intelligence system during a judicial process using a technique called “prompt injection.” The case, also reported by CNN Brazil, is more than a technical curiosity; it’s a stark warning for every company and developer integrating AI into critical workflows.
According to the OAB-PA’s decision, the lawyers inserted hidden instructions into an initial petition, aiming to influence a future analysis of the document by an AI system to favor their arguments. The OAB-PA described the alleged conduct as “serious procedural disloyalty,” undermining the credibility of the legal profession and the justice system. While the suspension is a preventive measure and not a final punishment, and the professionals will have the right to a full defense, the case already raises significant ethical and professional questions.
Do you have a question about AI security? You can reach us here:
Prompt Injection: Topping the OWASP LLM Top 10
The technique mentioned in the Brazilian case, “prompt injection” or “command injection,” is one of the most well-known and dangerous vulnerabilities in LLM-based systems. It’s no coincidence that it holds the top spot (LLM01) on the OWASP LLM Top 10 list. The essence of the attack is that an attacker provides a specially crafted input (a prompt) that overrides or manipulates the original developer-programmed instructions.
From an AIQ standpoint, the Brazilian case perfectly demonstrates this threat. Here, the legal petition itself was the Trojan horse. The hidden commands, embedded within seemingly innocuous legal text, could have instructed the AI to:
- Ignore the arguments of the opposing party.
- Assign greater weight to specific sections of the petition.
- Produce a biased summary or analysis favorable to the attackers.
In a corporate context, this means that any LLM application processing user input—be it a chatbot, a document summarizer, or an internal search tool—is potentially vulnerable. An attacker could manipulate the AI to leak confidential data, provide false information to decision-makers, or even execute unauthorized actions on their behalf.
Legal Ramifications: The EU AI Act and GDPR Perspective
Although the incident occurred in Brazil, it offers crucial lessons for European companies, especially with the full implementation of the EU AI Act and GDPR on the horizon.
From an AIQ standpoint, such an incident in Europe would carry severe legal consequences:
- EU AI Act: AI systems used in the administration of justice will, in most cases, be classified as “high-risk.” The Act imposes strict requirements on these systems regarding robustness, accuracy, and cybersecurity. A successful prompt injection attack would clearly demonstrate a lack of compliance, potentially leading to substantial fines.
- GDPR: A court petition almost certainly contains personal, and even special category, data. If a prompt injection attack causes the AI to access, modify, or leak such data without authorization, it constitutes a data breach. This would result in not only fines but also significant reputational damage.
The Brazilian lawyers were also accused of “litigating in bad faith.” This highlights that AI manipulation is not just a technical issue but a serious ethical and legal offense that erodes trust in digital processes.
The Key to Defense: LLM Red Teaming and Security Audits
How can organizations prepare for such attacks? In AIQ’s view, traditional IT security tools and methodologies are insufficient on their own against LLM-specific vulnerabilities. Defense must be multi-layered and built on proactivity.
The most effective method is LLM Red Teaming, where security experts simulate real-world attacks—including sophisticated prompt injection attempts—to uncover a system’s weak points before a real attacker can exploit them. This process helps to understand how the model reacts to unexpected, malicious inputs and allows for the implementation of necessary defense mechanisms (e.g., input filtering, output validation, instruction-following fine-tuning).
A comprehensive AI security audit covering all points of the OWASP LLM Top 10 is essential for any organization wishing to deploy artificial intelligence responsibly. The Brazilian case unequivocally proves that the risk of prompt injection is no longer theoretical. It has real, tangible consequences that endanger not only the system but also its users and the organization’s reputation. The question is no longer *if* it can happen, but *if* you are prepared for it.