Can AI errors shape the outcomes of the legal process?

The legal profession is no stranger to technological innovation, but the rapid adoption of AI tools has introduced unforeseen challenges. In Australia, a Victorian solicitor recently apologised to the court after submitting a list of fictitious authorities generated by an AI-powered research tool. The judge accepted the apology but referred the matter for investigation, highlighting broader issues of public policy and professional conduct.

Meanwhile, the research tool provider blamed the solicitor for failing to use the “… verification process, which sent the user the correct information just four hours after requesting it and well before appearing in court …”.[1]

We are seeing courts around the world are encountering AI-related errors, forcing the legal profession to confront questions of accountability and trust.

Can the legal system rely on technology without losing sight of human judgment? And who should bear the responsibility when AI goes wrong?

Courts’ patience has run out …

In a similar case in the UK, a taxpayer faced a penalty from the HMRC for the late payment of capital gains tax.[2] The taxpayer appealed citing ‘reasonable excuse’ based on her mental health and excusable ignorance of the law. To support her appeal, she submitted summaries of nine cases, supposedly provided by a friend in a solicitor’s office. However, the tribunal found that none of these cases were authentic.

The tribunal did not sanction the taxpayer and proceeded with the case as if the submissions had not been made. While it accepted the taxpayer’s ignorance about the cases being fabricated by AI and her inability to verify authorities, it rejected her claim of ignorance of the law regarding her late tax filing (under a recently amended rule) which her solicitor allegedly failed to advise her on.

Closer to home, earlier this year, a self-represented litigant in a sentencing hearing before the Supreme Court of the Australian Capital Territories failed to disclose the use of an AI tool in relation to a character statement submitted to the court. The Court inferred that AI had been used, observing that the language in the document was consistent with that of an AI-generated text. The Court found it ‘inappropriate’ that the character statement was prepared using AI and, as a result, had difficulty determining the weight, if any, that should be attributed to it.[3]

Lawyers are not afforded the same leniency as self-represented litigants. Returning to the case of the Victorian solicitor, neither the judge nor her associates could identify the list and summary of legal authorities submitted. While the solicitor claimed he did not intend to mislead the court, he admitted to not fully understanding how the AI tool worked and failing to verify the accuracy of the authorities. Despite the solicitor’s anxiety about the matter, the judge nevertheless considered it important to refer for investigations as a matter of public policy.

Even less fortunate were lawyers in New York who were sanctioned $5,000 for including six AI-generated fictitious case citations in a brief submitted to the court and making false and misleading statements to the court.[4] A Colorado lawyer faced a one-year suspension after he cited AI-generated case law in a motion submitted to a court in May 2023. The junior lawyer failed to verify the fictitious cases, and then failed to alert the court about the existence of the incorrect cases or to withdraw the motion.[5] In the Canadian case Zhang, [6] counsel mistakenly filed a notice of application referring to legal authorities fabricated by ChatGPT. The court held the counsel personally liable for the application cost and expense of the other party’s remediation research.

Reports indicate that Australian Courts frequently encounter AI-generated pleadings.[7] The Law Institute of Victoria and Law Society of NSW have published material on the responsible use of AI in line with the solicitors conduct rules. In these circumstances, we can expect even less understanding for future AI legal errors, particularly if court forms are equipped with a tick box to attest to the use of technologies in preparing submissions.

Is the solicitor the only party to blame?

Each solicitor is unequivocally responsible for their own work and court submissions, and must be familiar with authorities in their practice area. Any entirely unfamiliar AI-generated authorities should raise red flags. While some issues will be more challenging to detect – like in the UK case where AI-generated cases looked very similar to real cases – from a conduct perspective, there seems to be no justification for submitting false authorities to a court.

On the other and, like other professions, solicitors rely on commonly available technology for tasks such as secure file storage, sharing of confidential documents, document comparison tools, keyword searching large volumes of documents to comply with discovery orders, and conducting legal research. The lesson from this case, however, cannot be that solicitors should abandon these tools.

Secondly, despite an individual solicitor’s conduct responsibilities, employers who provide such technology for use in a workplace should consider how they can support staff better to avoid similar situations in future.

Thirdly, the service provider’s website may acknowledge that “… there is no room for errors…” when it comes to legal matters, but it will not accept any liability for the errors of its own product (but noting that in this case a subsequent human-led verification was provided). The provider will likely be protected by the service limitations and liability exclusions in its terms. Any output was likely delivered with a disclaimer such as “The above response is AI-generated and may contain errors. It should be verified for accuracy.”.

But should the provider’s liability be this limited? In the past, technology in testing phase was marked ‘beta’. With AI, customers are somehow expected to pay the full price for allegedly market-ready solutions which only perform to a beta standard.

AI risks and standards

Stanford researchers have discovered that even bespoke legal AI tools produce ‘hallucinations’ at 17% to 34% of cases.[8] A particular risk is in the tendency of AI to agree with the user’s incorrect assumptions expressed in the prompt. Even retrieval-augmented generation which splits the process into content retrieval and subsequent output generation, does not resolve the issue. Hallucinations come in the form of either incorrect output or output which is correct but supported by wrong citations.

The consequence are some real-life examples of AI risks in the legal profession, such as:

• Breach of confidentiality by using AI, as considered by OVIC in ordering a ban on generative AI tools for “Child Protection staff” in exercising their “official duties” at the Victorian Department of Families, Fairness and Housing, even for internally tenanted Microsoft 365 Copilot.[9]
• Case dismissed by court due to errors.
• Judge not giving due weight to evidence or submission suspected to be AI-generated.
• Apology given and hearing cost borne by the offending party.
• Reputational harm to solicitor and law firm.
• Lawyers sanctioned by the court for acting in bad faith.
• Professional conduct investigation by the legal profession authority.

Apart from voluntary frameworks, Australia currently lacks any AI standards mandated by the law (other than the recently introduced AI standards under online harms legislation).[10] The Voluntary AI Safety Standard released by the Department of Industry, Science and Resources will likely do little in the short term to improve AI providers’ practices. In practice, this means that law firms and lawyers purchasing AI-powered legal tools have no access to service performance evaluations and transparency around AI architecture. Providers continue to disclaim all liability and they benefit from the regulatory gap. This is understandable as they have to act in the best interest of their shareholders, and no lawyer would advise them to invite liability which is not due (certainly KHQ would not).

Regulatory frameworks around the world

In contrast to Australia’s voluntary framework, as part of its broader AI regulatory strategy, the European Union (EU) is rolling out a dual approach to managing AI risks: the AI Act (AIA) and the AI Liability Directive (AILD).

The AI Act requires providers of high-risk AI systems to deliver instructions on the characteristics, capabilities and limitations of performance of the high-risk AI system, including “ … the level of accuracy, including its metrics, robustness …” and “… specifications for the input data, or any other relevant information in terms of the training, validation and testing data sets used…”.[11] Using these criteria, the AIA categorises AI systems into four groups based on their potential for harm, with stricter requirements for higher-risk systems. High-risk AI systems are subject to the most rigorous regulatory obligations, including requirements for data governance, documentation, human oversight, robustness, accuracy, and security, as well as conformity assessments to demonstrate compliance.

The proposed AI Liability Directive introduces procedural changes that allow claimants to seek compensation for any loss or damage caused by “faulty” AI, even in the absence of a contractual relationship. In practice, this rule will allow a court to assume that non-compliance with an obligation caused the damage if the victim can show that the failure to meet the obligation is linked to the harm, and that there is a reasonable likelihood of a connection between the AI’s performance and the damage.[12]

In a similar vein, Canada is working to adopt a risk-based approach through its Artificial Intelligence and Data Act (AIDA). While still in development, AIDA aims to address the potential risks posed by AI by introducing criminal provisions that would penalize reckless and malicious uses of AI systems that could result in serious harm.[13] AIDA is expected to come into force no earlier than 2025, positioning Canada in alignment with the EU’s regulatory model.

However, other jurisdictions have been less proactive in regulating AI. The United States lacks a comprehensive federal approach to AI regulation, with various states and industry groups independently developing their own guidelines[14]. The United Kingdom has also been cautious, taking a more tentative stance with its regulatory approach, focusing on fostering innovation while providing limited guidance on AI governance[15]. Meanwhile, Singapore has opted for a ‘light-touch’ regulatory approach, favouring guidance and principles over hard laws. Singapore’s regulators aim to facilitate the safe and ethical use of AI without imposing stringent regulations, reflecting a more flexible and business-friendly stance compared to the EU’s robust regulatory framework.[16]

Mitigating AI risks in law

The key lesson is that lawyers must not rely on AI-generated outputs as substitutes for their own judgment and due diligence; they must verify the accuracy and reliability of the information provided. Lapses by lawyers in exercising proper care when utilising these AI tools can mislead the courts, jeopardise their clients’ interests, and ultimately weaken the rule of law.

Lawyers must be educated on the significant distinction between search and generative AI. One indexes information from various sources and helps us find it, while the other relies on its neural net to generate outputs based on unknown sources. We have developed a natural trust in search tools but must now teach ourselves to be sceptical of AI – even while paying for it.

Employers introducing AI tools in the workplace must do more, including:

• Introducing AI policies
• Providing workshops and training on common risks
• Strengthening their due diligence on AI providers before introducing them

Victorian Supreme Court guidance warns about AI-generated outputs being out of date and unaware more recent jurisprudence; incomplete and not contain all relevant points, inaccurate or incorrect, inapplicable to the jurisdiction or biased based on data which may over- or under-represent certain demographics or viewpoints.[17] However, there is more to do in the legal industry’s adjustment to AI.

At present, the solicitor may have no remedy against AI hallucinations, other than his or her own judgment and diligence. However, this defeats the purpose of having AI tools in the first place. While this conundrum is being solved by the legislature, employers introducing AI tools should push for more information, reassurance and responsibility from their providers.

Despite the significant risks discussed above, integration of AI into legal practice is feasible, but its implementation requires careful consideration. Lawyers must understand AI’s capabilities and limitations and exercise independent judgment based on their own skill. Further, solicitors must be prepared to explain AI-assisted work and provide additional supporting materials when necessary. Clients expect that the work performed by their solicitor reflects the solicitor’s expertise, judgment, and personal involvement. Thus, AI should be employed as a supplement to, not a replacement for, the legal services provided.

Conclusion

The integration of AI into legal practice offers tremendous potential but also significant risks. While AI can streamline tasks and improve efficiency, it is no substitute for human expertise and diligence.

Australia’s regulatory frameworks must evolve to address these challenges. The European Union’s AI Act and Canada’s Artificial Intelligence and Data Act provide compelling examples of risk-based governance. Australia must not lag behind; clear standards are essential to protect the integrity of the legal profession.

Ultimately, while AI will never replace the judgment and skill of a solicitor, it is a tool that can augment the profession—if used responsibly.

This article was written by Alex Dittel (Principal Solicitor) and first appeared in the February edition of the Internet Law Bulletin (Vol 27, No 1 & 2) published by LexisNexis.

Want Data Privacy, Cyber & Digital updates delivered straight to your inbox? Click here to subscribe. 

[1] As above.

[2] Felicity Harber v The Commissioners for HMRC ([2023] UKFTT 1007 (TC)).

[3] DPP v Khan [2024] ACTSC 19 at [43].

[4] Roberto Mata v Avianca Inc, 22-cv-1461 (PKC), US DC Southern District of New York.

[5] www.cbsnews.com/colorado/news/coloradolawyer-artificial-intelligence-suspension/

[6] Zhang v. Chen, 2024 BCSC 285

[7] People are citing non-existent cases and giving rambling statements in court. Judges blame AI, Crikey, 13 September 2024 (link)

[8] AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries, Stanford University, May 23, 2024 (link).

[9] Investigation into the use of ChatGPT by a Child Protection worker, OVIC, 3 September 2024 (link)

[10] Online Safety (Basic Online Safety Expectations) Determination 2022 amended in 2024.

[11] Article 13, AI Act (link).

[12] AI Liability Directive, Article 4(1).

[13] Innovation, Science and Economic Development Canada, ‘Artificial Intelligence and Data Act (AIDA): Companion Document’ (Web Page, 2024)

[14] Brookings Institution, ‘The EU and US Diverge on AI Regulation: A Transatlantic Comparison and Steps to

Alignment’ (Web Page, 4 May 2023)

[15] AI regulation: a pro-innovation approach, Department for Science, Innovation and Technology (UK), 2023

[16] Personal Data Protection Commission, Model Artificial Intelligence Governance Framework (Report, 2020)

[17] Guidelines for litigants: responsible use of artificial intelligence in litigation, Supreme Court of Victoria (link)