1,006 court filings exposed AI hallucinations judges never caught

1,006 court filings and counting: the AI hallucination crisis nobody fixed

A lawyer in New York submitted a brief packed with case citations that looked perfectly legitimate. Six of those cases never existed. The judge fined him $5,000, and his career became a cautionary tale retold at every legal tech conference since 2023. But here is what most retellings leave out: that lawyer was not an outlier. A running database now tracks over 1,006 legal filings worldwide containing fabricated AI-generated citations, and the number grows weekly.

The uncomfortable question is not whether AI belongs in courtrooms. It is already there. The question is what happens when the technology that promises to democratize justice quietly encodes the opposite.

The 300% efficiency gain courts cannot ignore

In Argentina, the Prometea AI assistant helped Buenos Aires prosecutors jump from 130 cases per month to nearly 490, a productivity leap that would require tripling staff the old-fashioned way. Brazil’s VICTOR system, deployed at the Supreme Court, reduced the time to evaluate appeals from 44 minutes per case to seconds. China’s Shenzhen Intermediate People’s Court became the first to systematically integrate a large language model into judicial reasoning in June 2024, training it on two trillion characters of legal text.

These are not pilot programs. They are operational systems shaping real verdicts that affect real people’s freedom, property, and fundamental rights.

Courts in at least 12 countries now use AI in some form of decision-making. The efficiency argument is settled: AI slashes backlogs. But efficiency is a dangerous metric when liberty is at stake.

The bias problem that speed cannot solve

When researchers at PMC examined AI bias in adjudication, they found something that should alarm anyone who has ever stood before a judge: the historical data these systems learn from already contains decades of documented racial, socioeconomic, and geographic bias. Training AI on biased decisions does not eliminate prejudice. It automates it, at scale, and wraps it in the authority of algorithmic objectivity.

The Netherlands learned this the hard way. Its SyRI system, designed to detect welfare fraud, was struck down by a Dutch court for violating the European Convention on Human Rights. The system disproportionately targeted low-income neighborhoods with high immigrant populations. The technology worked exactly as designed; the design was the problem.

Colombia’s experience tells a more nuanced story. When Judge Juan Manuel Padilla openly used ChatGPT to help draft a ruling in a healthcare case involving an autistic child, the Springer Nature study that examined the case found that the AI-assisted decision was substantively reasonable. But Colombia’s Constitutional Court, in ruling T-323/24, still mandated transparency requirements, recognizing that even reasonable outcomes do not justify opaque processes.

The regulatory void that 73% of judicial operators see

UNESCO published its first comprehensive guidelines for AI in courts and tribunals in 2025, gathering contributions from 41 countries. The document acknowledged what practitioners already knew: most jurisdictions have no binding rules for how judges can use AI tools.

France stands alone as the most aggressive regulator. Since 2019, Article 33 of its Justice Reform Act criminalizes the use of AI to analyze or predict judicial decisions, with penalties of up to five years in prison. The law exists not because France fears efficiency, but because it fears a world where litigants game the system by predicting which judge will rule in their favor.

Germany and the EU AI Act take a different path, classifying judicial AI as "high-risk" and requiring conformity assessments without outright banning it. Most other countries? They have no framework at all.

What this means for anyone who might face a judge

The gap between AI’s capability and its governance in courts is not closing. It is widening. Every month, new systems go live in new jurisdictions while regulators draft position papers.

If you have ever assumed that a court ruling reflected a human judge’s careful reasoning, that assumption now requires verification. Cambridge researchers warn that uncritical reliance on generative AI erodes the deep legal reasoning that depends on careful reading and analogical thinking: precisely the skills that distinguish a judgment from a calculation.

The 1,007th hallucinated filing is being drafted somewhere right now. The question is whether the judge reviewing it will know the difference, and whether anyone requires them to check.