AI – what if there is too much artificial and not enough intelligence? 

By this stage, no doubt even the uncontacted Mascho Piro tribe in Peru have heard about the rise of artificial intelligence (AI), and its potential disruption to virtually all industry sectors. Closer to home, listeners of a Sydney radio station were recently made aware that the radio show’s morning show was being run by an AI DJ for six months without anyone apparently noticing.  

Risks 

Disruption in the legal sector is no exception, and the courts, lawyers, and clients are all scrambling to understand how best to use AI, and the risks in doing so. 

Those risks have been brought into stark relief for some practitioners, with lawyers in both Victoria and New South Wales being referred to the relevant disciplinary bodies for their use of AI in preparing and submitting case summaries to the court.  

In the Victorian example, the lawyer used an AI tool provided by a legal research platform to provide a list of cases, all of which were “hallucinations” by the AI – i.e. entirely fictitious. In NSW, the lawyer used the freely available ChatGPT AI, with similarly artificial results. Both lawyers have had their names suppressed, but we would expect any future occurrences would not be treated with similar levels of confidentiality. 

In both those examples, the court noticed the fictitious authorities during the running of the matter.  In the US, there has already been one example of a court issuing an order based in part on fake cases.  The lawyer doubled down on appeal, and relied on further fake cases to support the earlier order, and even sought costs, citing one of the new hallucinated cases in support.  The Georgia Court of Appeals stopped the hallucinating merry-go-round at that point, but it may only be a matter of time before a hallucinated case gets through even that level of scrutiny. 

As at the date of this article, a publicly available database has identified that AI created ‘hallucinated’ content has appeared in 17 Australian proceedings, noting that this only covers proceedings where the Court identified the false content, and referred to it in a decision. 

Australian courts are releasing practice notes and other guidelines governing the use of AI. 

Judicial approach to use of AI 

The Victorian Supreme Court has adopted a relatively progressive approach to AI use, acknowledging the benefits of AI in litigation and encouraging AI use to achieve greater efficiencies.  

Key risks associated with AI have also been acknowledged however, including that in its current form it is prone to hallucinations or may produce out-of-date information. Ultimately, the Court’s view appears to be that AI can be used, provided it does not breach the obligations that litigants and lawyers must comply with. That is, information generated by AI tools needs to be scrutinised, assessed and reviewed to ensure that the information is accurate and delivered to the Court in good faith. The Court has refrained from banning or setting restrictions in respect of AI use; responsibility lies with litigants and legal practitioners to forensically assess AI output in preparation of each case.   

In contrast stands the Supreme Court of New South Wales’ position. In November 2024, the Court issued a Practice Note regarding AI use. It appears that the NSW Supreme Court has taken a much more conservative and sceptical view compared to its Victorian counterpart. While participants in the NSW jurisdiction are permitted to use Generative AI for drafting general documents (such as summaries), legal research and to assist with formatting and grammar, the Court has specifically prohibited the use of AI to:  

• draft affidavits and witness statements;  

• generate expert reports; and  

• upload confidential or restricted information to public AI databases. 

The Federal Court of Australia is currently running a public consultation process on the use of AI ahead of publishing its own Practice Note. In the interim, the Federal Court’s holding position is similar to that of Victoria’s; AI use is permitted if its use is consistent with lawyers’ existing obligations and its use is disclosed to the Court upon request.  

In our view, we doubt that AI in its current iterations could be used for the preparation of affidavits or witness statements and still comply with a lawyer’s obligations, even if doing so is not explicitly prohibited in the jurisdiction.   

Fundamentally, an AI (or more specifically, a Large Language Model) is just a very sophisticated text predictor.  It has been trained on huge datasets in order to produce natural sounding language.  However, unless it is given extremely detailed “prompts” (which would largely remove the efficiency benefits of using an AI) it cannot generate the witness’ own words or recollection.   

Some may argue this is not fundamentally different to a lawyer carefully crafting a witness statement (leading to the notorious examples of witnesses who do not have a strong grasp of spoken English but were apparently able to give their lawyer detailed instructions about complex legal arrangements), but in that instance there is at least an officer of the court directly preparing the statement.  An AI owes no duties to the court and is incapable of having regard to any ethical considerations of improperly putting words in the mouth of a witness. 

On that basis, the approaches of the Victorian and New South Wales Courts may not be that different in practice. 

The AI genie is well and truly out of the bottle, but as Jafar discovered when tricked by Aladdin, you have to be careful what you wish for when using it.  

This article was originally written by Jordan Diamantopoulos and Paul Welling.

Want Litigation & Dispute Resolution updates delivered straight to your inbox? Click here to subscribe.