A recent Supreme Court case in India has shifted the focus of AI risks from misinformation and deepfakes to a more fundamental question: who is responsible when AI-generated information influences a decision and turns out to be incorrect? The issue arose after the Supreme Court set aside an order by the National Company Law Tribunal (NCLT) that relied on non-existent legal precedents allegedly generated by AI, highlighting the growing risks of using AI outputs in critical decision-making.
As organizations increasingly rely on generative AI for legal research, financial analysis, compliance, healthcare, and business operations, accountability becomes more complex. Experts point out that if AI fabricates financial data used in lending, produces inaccurate legal references, or generates flawed medical summaries, responsibility cannot simply be assigned to the AI itself. Instead, the accountability may rest with the human users, organizations deploying the technology, or developers, depending on how the AI was used and whether adequate oversight was in place.
The article underscores that AI should support—not replace—human judgment, particularly in high-stakes decisions. Organizations are encouraged to verify AI-generated outputs, maintain clear governance frameworks, and ensure that qualified professionals review AI-assisted decisions before they are acted upon. Human oversight remains essential to minimize errors and prevent harmful consequences arising from AI "hallucinations" or misinformation.
The case also strengthens the argument for dedicated AI regulations in India and elsewhere. As AI becomes embedded in decision-making across industries, policymakers are increasingly focusing on legal accountability, transparency, and governance to ensure that organizations deploying AI remain responsible for its outcomes. The broader consensus is that while AI can enhance productivity and decision-making, the ultimate responsibility for critical decisions must continue to rest with humans and the institutions that deploy these systems.
