“The most dangerous lie in the age of AI is not the one a machine invents—it is the one a human stops questioning.”
— Adv. Tarun Choudhury
Introduction: When Artificial Intelligence Starts Inventing the Law
Artificial intelligence has entered the legal profession with extraordinary speed. Lawyers now use AI to research cases, summarise judgments, identify legal issues and prepare drafts that once required hours of manual work.
But a dangerous question has emerged: What happens when artificial intelligence does not merely make a mistake, but invents the law itself?
India’s Supreme Court has now provided a powerful answer.
In Vijay Ghanshyam Gadiya v. Union of India & Anr., 2026 INSC 947, decided on 2 September 2026, the Supreme Court set aside a customs adjudication involving a penalty of ₹425,27,99,100, after finding that the adjudicating authority had relied upon case law that was non-existent, carried fake citations, or did not support the legal propositions attributed to it. The Court described the material as appearing to be an AI hallucination.
This was not simply a warning against bad legal research.
It was a warning about the integrity of adjudication itself.
What Happened in Vijay Ghanshyam Gadiya?
The Additional Commissioner of Customs, Surat, imposed the enormous penalty under Section 114 of the Customs Act, 1962, alleging misdeclaration of natural diamonds as laboratory-grown diamonds to obtain a lower tariff.
The Gujarat High Court dismissed the challenge on 20 January 2026.
When the matter reached the Supreme Court, the appellant contended that several judgments and articles relied upon in the original adjudication order had been generated using AI.
The Supreme Court did something particularly significant.
It independently checked the authorities.
The Court found that some cases were non-existent or had fake citations. It further discovered that some genuine judgments did exist, but did not lay down the propositions for which they had been cited.
The Court consequently found the reliance on such dubious material fatal to the sustainability of the order.
Supreme Court Directions in Vijay Ghanshyam Gadiya
- Set aside the Gujarat High Court’s order;
- Set aside the original customs adjudication order;
- Revived the proceedings for fresh adjudication;
- Directed that the matter be heard by another officer of the same rank; and
- Left it to the appointing authority to consider appropriate action against the author of the original order.
Importantly, the Supreme Court did not finally decide the underlying customs dispute. The case was remanded for a fresh decision.
That distinction matters.
The Real Constitutional Issue: Can Fiction Become Law?
The deeper problem is not that a citation was incorrect.
Lawyers have always made mistakes.
The extraordinary danger presented by AI hallucinations in courts is different: an AI system can manufacture an entirely plausible legal universe.
It can produce:
- A fictitious judgment;
- A real case with the wrong citation;
- A real judgment with a fabricated quotation;
- A genuine case containing a proposition it never decided;
- A non-existent paragraph number; or
- An apparently authoritative line of precedent that never existed.
The output may look perfectly professional.
That is precisely what makes it dangerous.
A court cannot determine rights on the basis of authorities that exist only inside a machine-generated answer.
Pooja Ramesh Singh: The Supreme Court’s Foundational AI Judgment
The latest decision must be read alongside the Supreme Court’s earlier judgment in Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd., 2026 INSC 668, decided on 2 July 2026.
There, the Supreme Court confronted a tribunal decision relying upon non-existent, fake and hallucinated AI-generated material as precedent.
The Court set aside the NCLT and appellate decision and made an extraordinary declaration: courts must adopt “zero tolerance” towards producing, citing or using AI-generated precedents without verification. It also characterised an advocate’s unverified citation of such material as misconduct and a judge’s reliance upon it as a serious lapse.
The Court went further: a decision relying upon such fake or hallucinated material cannot be sustained because the sanctity of adjudication has been compromised.
Key Legal Principle on AI and Adjudication
This establishes a crucial legal principle:
AI can assist adjudication, but it cannot become the source of legal authority.
AI May Assist—But It Cannot Sit in the Judge’s Chair
The language used by the Supreme Court in Vijay Ghanshyam Gadiya is likely to become one of the most memorable judicial observations on AI:
“Assistance can never be substituted for adjudication.”
The Court added that AI may serve as “training wheels”, but putting AI in the “pilot’s seat” would be imprudent and dangerous.
This is a much more nuanced approach than simply banning AI.
The Supreme Court expressly clarified that the judgment does not concern the rightful use of AI. The objection is to presenting or relying upon fake or hallucinated material as genuine legal precedent.
That distinction should guide the future of AI-assisted legal practice.
Can Lawyers Use ChatGPT for Legal Research?
Yes—but never as the final authority.
AI can be extremely useful for:
- Generating research questions;
- Identifying potentially relevant cases;
- Summarising lengthy judgments;
- Comparing legal concepts;
- Organising research;
- Preparing preliminary drafts; and
- Identifying statutes and legal issues for further investigation.
Proper AI-Assisted Legal Research Workflow
AI → Primary Source → Verification → Legal Analysis → Filing
It must never become:
AI → Copy → Paste → Court
Every case cited in a pleading should be independently verified.
What Lawyers Must Verify Before Citing AI-Assisted Research
| Verification Point | What the Lawyer Should Confirm |
|---|---|
| Case Existence | The case exists. |
| Citation | The citation is correct. |
| Court and Date | The court and date are correct. |
| Quoted Passage | The quoted passage actually appears in the judgment. |
| Legal Proposition | The case supports the proposition for which it is cited. |
| Current Validity | The precedent has not been overruled or materially distinguished. |
This is not merely good practice.
After Pooja Ramesh Singh, it is increasingly part of the legal profession’s duty of verification.
The Global Position: India Is Not Alone
The significance of India’s Supreme Court approach becomes clearer when compared with developments elsewhere.
United States
The famous Mata v. Avianca litigation demonstrated the consequences of submitting fictitious AI-generated authorities. Lawyers were sanctioned after non-existent cases generated through ChatGPT were presented to the court.
The American lesson was clear: AI does not transfer professional responsibility away from the lawyer.
United Kingdom
The UK’s judicial guidance expressly warns about AI hallucinations, including incorrect or misleading information, and emphasises that judicial office holders retain personal responsibility for material produced in their name.
The UK regulatory position has now become even stronger. The Solicitors Regulation Authority’s August 2026 warning states that solicitors remain accountable for AI-assisted work and must ensure that cited authorities are genuine, verifiable and relevant. It specifically warns that relying on AI output is not a defence to placing false authorities before a court.
Canada
Ontario’s Superior Court of Justice has expressly directed lawyers and litigants to verify AI-generated legal information against authoritative sources. It recognises hallucinations as potentially involving non-existent cases, mischaracterised case law and fabricated quotations.
The court also identifies possible consequences including cost orders, adjournment, dismissal, contempt proceedings and referral to the Law Society.
British Columbia’s Provincial Court issued similar guidance in July 2026, telling lawyers and litigants to verify AI-generated cases and legislation and, where possible, provide hyperlinks to authorities.
Australia
Australian courts have likewise confronted fabricated authorities and emphasised that lawyers cannot delegate their responsibility for accuracy to an AI system.
Singapore
Singapore has adopted a similar controlled-use philosophy: AI may assist legal work, but lawyers remain responsible for the accuracy and authenticity of material submitted to court.
The Next AI Problem Is Bigger Than Fake Case Law
The next stage of the debate will not concern hallucinated judgments alone.
It will involve confidentiality, privilege, bias, explainability and accountability.
A lawyer may unknowingly expose confidential client information by uploading pleadings, contracts, medical records or privileged communications into an inappropriate AI system.
The UK’s 2026 regulatory warning expressly identifies confidentiality and legal professional privilege as major AI risks.
Therefore, the future AI protocol for lawyers must contain two separate safeguards:
- First: Verify the law.
- Second: Protect the client.
An AI system that gives a correct answer after receiving confidential privileged material can still create a serious professional problem.
India’s Emerging Judicial AI Framework
The Supreme Court’s approach is also consistent with its institutional development.
The Supreme Court has published draft Regulations for Use of Artificial Intelligence in Courts, 2026. The draft framework expressly addresses:
- Human primacy;
- Judicial independence;
- Rule of law;
- Transparency;
- Explainability;
- Accountability;
- Auditability;
- Privacy;
- Proportionality;
- Data integrity; and
- Cybersecurity.
This is significant because it demonstrates that India’s position is not anti-AI.
It is pro-responsible-AI.
The objective is not to prevent technology from entering courts.
The objective is to ensure that technology never displaces judicial responsibility.
A Proposed Five-Point AI Rule for Every Lawyer
In my view, every advocate using AI should adopt the following professional rule:
- Never cite an AI-generated case without finding the original judgment.
- Never quote a judgment without reading the relevant passage.
- Never assume that a correct-looking citation means a correct proposition.
- Never upload confidential client material into an AI platform without appropriate safeguards.
- Never tell the court that “AI made the mistake” as though that removes professional responsibility.
The machine may have generated the error.
The lawyer chose whether to rely upon it.
Can an AI-Hallucinated Authority Destroy a Judgment?
The answer emerging from the Supreme Court is yes, where the integrity of adjudication has been materially compromised.
But Vijay Ghanshyam Gadiya should not be misunderstood as creating a rule that every minor AI-related mistake automatically nullifies a judgment.
The important question is whether reliance upon fabricated or unreliable material has contaminated the adjudicatory process.
In Pooja Ramesh Singh, the Court adopted an exceptionally strict position because judicial decision-making cannot be permitted to rest upon fictional authorities.
Emerging Doctrine on AI-Generated Falsehoods
Thus, the emerging doctrine is better stated as:
The more deeply AI-generated falsehood penetrates the reasoning process, the greater the threat to the validity of the adjudication.
Conclusion: The Human Must Remain Accountable
The Supreme Court’s decisions in Pooja Ramesh Singh and Vijay Ghanshyam Gadiya may eventually be remembered as foundational cases in India’s law of artificial intelligence.
Their importance extends far beyond ChatGPT.
They establish a principle that should govern the entire AI-assisted justice system:
Technology can assist legal reasoning. It cannot inherit legal responsibility.
- A lawyer remains responsible for the authorities placed before the court.
- A judge remains responsible for the judgment delivered.
- A tribunal remains responsible for the integrity of its adjudication.
- And a court cannot surrender its constitutional function to a probabilistic machine capable of inventing authorities that never existed.
The legal profession should therefore not fear AI.
It should discipline AI.
The future courtroom will almost certainly use artificial intelligence. The question is whether lawyers and judges will use it as a powerful assistant—or allow it to become an unaccountable decision-maker.
The Supreme Court has already supplied the answer:
AI may be the tool. Humans must remain responsible for the law.
And that may ultimately be the most important rule governing AI hallucinations in courts.
Citation: Vijay Ghanshyam Gadiya v. Union of India & Anr., 2026 INSC 947; Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd. & Anr., 2026 INSC 668.
Need Expert Legal Guidance?
AI can assist legal research, but it cannot replace experienced legal judgment. If your case involves complex litigation, disputed precedents, Supreme Court matters, legal research, or concerns arising from AI-generated legal content, professional legal guidance is essential.
Adv. Tarun Choudhury
Supreme Court Advocate | 25+ Years of Legal Experience
📞 Call: 9650499965
💬 WhatsApp: 8851978611
Get experienced legal guidance before relying on AI-generated legal advice, case law, or documents.
Frequently Asked Questions: AI Hallucinations in Courts
1. What Are AI Hallucinations in Courts?
AI hallucinations in courts occur when artificial intelligence generates false or fabricated legal information, such as non-existent judgments, incorrect case citations, invented quotations, or legal propositions that are not supported by the cited authority.
The Supreme Court of India has warned that such unverified AI-generated material can seriously undermine the integrity of judicial proceedings.
2. Can an AI-Hallucinated Case Law Invalidate a Court Judgment in India?
Yes, AI-hallucinated case law can lead to a judgment or adjudication order being set aside where reliance upon fabricated or incorrect authorities materially compromises the decision-making process.
In Vijay Ghanshyam Gadiya v. Union of India, 2026 INSC 947, the Supreme Court set aside the impugned adjudication after finding serious problems with the authorities relied upon.
3. Can Lawyers Use ChatGPT and Other AI Tools for Legal Research?
Yes. Lawyers can use ChatGPT and other AI tools for legal research, including identifying potential authorities, summarising judgments and organising legal issues.
However, AI-generated case law must be independently verified against the original judgment and authoritative legal databases before being cited in court.
AI assistance does not remove a lawyer’s professional duty to verify legal authorities.
4. Who Is Legally Responsible When AI Generates a Fake Case Citation?
The lawyer, judge or adjudicating authority using the AI-generated material remains responsible for verification and its use.
An AI system cannot assume professional or judicial responsibility.
The Supreme Court’s approach in Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd., 2026 INSC 668, emphasises that courts and lawyers must not rely upon fake or hallucinated precedents without verification.
5. What Did the Supreme Court of India Say About AI Hallucinations and Legal Judgments?
The Supreme Court has adopted a zero-tolerance approach to unverified AI-generated precedents.
It has emphasised that AI may assist legal research and adjudication but cannot substitute human judicial reasoning.
In Vijay Ghanshyam Gadiya, the Court warned that AI may serve as a useful tool but that placing it in the “pilot’s seat” of adjudication would be imprudent and dangerous.
The emerging principle is that AI can assist justice, but human lawyers and judges must remain legally accountable.
Key Takeaways: AI Hallucinations in Courts
- AI hallucinations in courts are now a serious legal concern. Artificial intelligence can generate fake judgments, incorrect case citations, fabricated quotations and legal propositions that do not exist in the cited authority.
- The Supreme Court of India has issued a strong warning against unverified AI-generated case law. In Vijay Ghanshyam Gadiya v. Union of India & Anr., 2026 INSC 947, the Court set aside a customs adjudication involving a ₹425.27 crore penalty after finding serious problems with the legal authorities relied upon.
- The Supreme Court’s earlier decision in Pooja Ramesh Singh is a foundational authority on AI-generated fake precedents. In Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd., 2026 INSC 668, the Court adopted a zero-tolerance approach towards citing or relying upon AI-generated authorities without proper verification.
- AI cannot replace judicial reasoning. The Supreme Court has made it clear that AI may assist lawyers, judges and tribunals, but human beings must remain responsible for legal analysis and adjudication.
- Lawyers remain professionally responsible for AI-assisted legal research. Saying that “AI generated the citation” does not excuse a lawyer from verifying the case, citation, quotation, paragraph number and legal proposition before relying upon it in court.
- Every AI-generated legal authority must be independently verified. Lawyers should check the original judgment, authoritative database, court, date, citation, relevant paragraph and whether the precedent remains good law.
- AI hallucinations can have serious consequences. A fabricated precedent can influence judicial reasoning, affect substantial financial liabilities and potentially undermine the fairness and legitimacy of an adjudicatory process.
- The problem is global, not merely Indian. Courts and legal regulators in the United States, United Kingdom, Canada, Australia and Singapore have also confronted fabricated or hallucinated AI-generated case law.
- Singapore, Australia, Canada and the United States demonstrate an emerging international principle: lawyers may use AI as a research tool, but they remain responsible for the accuracy and authenticity of material submitted to courts.
- AI should follow a human-in-the-loop model. The safest legal workflow is:
| Stage | Legal AI Workflow |
|---|---|
| 1 | AI assistance |
| 2 | Primary-source verification |
| 3 | Legal analysis |
| 4 | Human review |
| 5 | Court filing |
- AI should never be treated as a primary legal authority. ChatGPT and other generative AI tools can help identify potentially relevant law, but the actual judgment, statute or authoritative legal source must be examined before reliance.
- Confidentiality is another major AI risk for lawyers. Legal professionals must be careful when entering privileged or confidential client information into AI systems.
- India is moving towards responsible judicial AI regulation. The Supreme Court’s proposed Regulations for Use of Artificial Intelligence (AI) in Courts, 2026 emphasise human primacy, judicial independence, accountability, transparency, auditability, privacy and data integrity.
The emerging legal principle is simple: AI can assist justice, but it cannot assume legal responsibility.
Summary
AI hallucinations in courts are transforming the rules of legal research and professional responsibility.
India’s Supreme Court has warned that lawyers and judges cannot rely on fake or unverified AI-generated case law.
The judgments in Pooja Ramesh Singh and Vijay Ghanshyam Gadiya establish an important principle for the AI era: artificial intelligence may assist legal research and adjudication, but every legal authority must be independently verified and final legal responsibility must remain with human lawyers and judges.
