“Some of these expressions are – GenAI, Human-in-the-Loop, Black Box, Hallucination, Risk-Scoring, Algorithmic Decision-Making.”
In 1989, Sir Tim Berners-Lee invented the World Wide Web[2] (www), and in the early 90s, the www reached our country. With the invention came new terminologies that became part of our lives. The terms – ‘Computer’, ‘Network’, ‘Cyber Café’ became launchpad for ‘electronic form’, ‘Data’, ‘Digital Signature’, ‘Cyber Security’, to name a few. In 2000, our Parliament enacted the Information Technology Act recognizing them in the eyes of law.
The sensorium encompassing of Artificial Intelligence (AI) has again introduced new expressions that are likely to enter our lexicon. The preliminary draft of ‘Regulations for Use of Artificial Intelligence in Courts 2026’[3] (‘Draft Regulations’) prepared under the aegis of the Artificial Intelligence Committee, Supreme Court of India, defines the new terms that judges and legal practitioners must know to better understand the technology. Some of these expressions are – GenAI, Human-in-the-Loop, Black Box, Hallucination, Risk-Scoring, Algorithmic Decision Making.
Generative AI (‘GenAI’) means an AI system that can generate new content, such as draft of a pleading, report, an image, an audio clip, a video, etc. It may do so either on the basis of training given to it by the developers, or data and prompt given by a user; and watching it accomplish these tasks when the hour-marker on the clock hasn’t moved is parallelly fascinating & humbling. In our legal field, GenAI systems may be used to create a contract or draft a petition/reply, written submissions or list of dates and even a judgment. It can create translations, summaries and even reports. A deeper, detailed prompt has the potential to generate matured content. The Draft Regulations proposes that no GenAI generated content shall be filed or produced before any court without mandatory disclosure of its origin. This will bring the use of AI systems in courts at par with the globally existing norms in the U.S.A., E.U., Singapore, to name a few. Under the ‘Guide on the Use of Generative Artificial Intelligence Tools by Court Users’[4] issued by Singapore, ‘Generative AI tools’ are described as tools that generate text, image, audio and video output in response to prompts from users, using patterns and correlations identified in the data on which they were trained.
These abilities make an AI system a useful assistant. But every case has varied elements compared to another. While AI system may be trained on large volumes of data, but none equates with the ability of an advocate or a judge to give a tailored fit to each case. For this reason, blind reliance on an AI system breaches administration of justice. To ensure that the rule of law is upheld the Draft Regulations declare the use of artificial intelligence in court processes to remain subservient to human judgment and judicial authority, and that every AI system shall function in an assistive capacity. It further introduces the concept of keeping ‘Human-in-the-Loop’, in which outputs of GenAI are subject to mandatory human review, supervision, and verification. Article 14 of the European Union Artificial Intelligence Act 2024[5] deals with human oversight and provides that high-risk AI systems must be designed to enable humans to oversee, understand, and intervene on the operation of the AI system, including via an emergency stop procedure. Humans must be able to understand the system’s capacities and limitations, detect and address biases and errors, and override or interrupt the system. The United Kingdom Judiciary Guidance 2025[6] embodies similar principle without using the exact phrase. The UK Judiciary Guidance holds judicial officers personally responsible for ‘material which is produced in their name’ and that ‘judges must always read the underlying documents’. The Singapore Court also released a ‘Guide on the Use of Generative Artificial Intelligence Tools by Court Users’, which establishes that in case of using AI, the responsibility for accuracy and relevancy lie with the user.
One may wonder about the transparency in functioning of these enormously powerful systems. For instance, prompt an AI System to find judgments on limited liability clauses in a contract and varied results on different AI Systems showing some judgments and leaving out other relevant case laws raises concern surrounding unclear steps taken to perform the task by AI system in the background. Similarly, in a contract review AI system may raise red flags without sufficient logic to it. Acknowledging the opaque functioning of the AI systems, the Draft Regulations define ‘Black Box’ referring to an AI system which employs deep learning techniques and whose internal processes and decision-making logic are not transparent or capable of being explained by reference to identifiable rules. An AI System may also ‘Hallucinate’ i.e. include fake amendments to a statute, quote wrong provision in law, produce non-existent decisions or quote genuine citations but forge portions of it. The United Kingdom Judiciary Guidance 2025 contemplates that AI tools may “hallucinate” which includes making up fictitious cases, citations or quotes, or refer to legislation, articles or legal texts that do not exist; provide incorrect information regarding the law or how it might apply and make factual errors.
Using an AI system to predict the risk of a person committing a criminal offence based on their profiling, past records, and personality traits has brought out biases and opacity embedded in such systems. While the bias is a result of the data used to train the AI system, however, opacity remains unexplained even by the developers at times. In Draft Regulations, ‘Risk Scoring’ means using an AI system to assign a score to an individual to estimate the probability of that person engaging in a crime, committing an offence, or failing to appear before a court in the future and Regulation 20 prohibits use of AI for risk scoring. Similarly, Article 5 of the European Union Artificial Intelligence Act 2024 prohibits using an AI system for making risk assessment of a person in order to assess or predict the risk of such person committing a criminal offence. The exercise of decision making can be carried out using algorithms embedded in an AI system. ‘Algorithmic Decision Making’ strikes at the root of justice dispensation raising issues of transparency, fairness, equality and accuracy. The Draft Regulations prohibit arriving at judicial outcome (including any judgment, order, or finding of fact or law) through Algorithmic Decision-Making alone or solely on the basis of AI-generated information. The European Union Artificial Intelligence Act 2024 refers to use of AI systems to detect decision-making patterns or its use in administration of justice by judicial authority in researching and interpreting facts and the law and in applying the law to a concrete set of facts to be a High-Risk AI System which shall necessarily have risk management systems and places humans in the position of overseeing.
These are only some of the instances which place human primacy at the heart of the use of AI systems in the legal field. AI systems have the potential to generate synthetic content indistinguishable from human-generated. The increasing capabilities of the AI systems impact integrity and trust and raise new risks of misinformation and manipulation. These issues vociferate a pressing need for urgent framing of guidelines for the use of AI systems for all the courts and advocates across our country.
Over years, these jargons will become words of common parlance as well. Like the electric batteries in cars, rightful use of AI is going to silently propel the speed of our advocates and judges alike. Given the expansive capabilities of the AI systems, they are surely to stay which makes it imperative for all the stakeholders in the legal field in our country to familiarize themselves with these new expressions. AI is going to expand its reach in the legal field as the technology spreads and over time, it will be given space in the Statutes. While regulation or guidelines are framed, the provisions and intent behind the Draft Regulations may be voluntarily incorporated in practice and adjudication across our country.
[1] Lalltaksh Joshi, Advocate-On-Record
[2] A short history of the Web – Home | CERN
[4] guide-on-the-use-of-generative-ai-tools-by-court-users.pdf
[5] Regulation – EU – 2024/1689 – EN – EUR-Lex
[6] Artificial Intelligence (AI) Guidance for Judicial Office Holders