People are deceiving the justice system with AI

6 hours ago

Lawyers in Brazil attempted to deceive the justice system by hiding instructions in white text within petitions, targeting the AI system Galileu used by the court. The AI detected the attempt, leading to sanctions.
This 'prompt injection' technique is not science fiction but an expected development, with legal frameworks like Spain's already considering AI use in courts.
Prompt injection involves inserting hidden prompts to manipulate AI systems, and similar tactics have been used in education and hiring to detect or influence AI behavior.
The legal world is particularly vulnerable due to the high volume of documents and the state's authority, with prior cases of lawyers using AI to fabricate case law.
Galileu assists judges by drafting rulings but requires human review; the fraud aimed to eliminate evidence before it reached the judge.
Other Brazilian courts have found similar attempts, including a prompt ordering the AI to grant all requested measures.
The real danger is 'invisible fraud'—altering the process of reading and interpretation, not just falsifying content.
Risk extends to law firm document exchanges, raising concerns about professional ethics and the fine line between zealous defense and deception.

Hasty Briefsbeta