Benchmarking Legal AI
A small newsletter on legaltech
AI
A legal benchmarking study was released last week comparing major legal-specific GenAI platforms [LegalTech Hub | Artificial Lawyer | Legal IT Insider | Vals Legal AI Report]
Last month, the DC Court of Appeals released an opinion in an animal cruelty case where both the majority and the dissent referred to using ChatGPT as part of their decision-making process [Dentons client alert | Opinion]
From The Daily Beast: “The Los Angeles Times removed its new AI-powered ‘insights’ feature from a column after the tool tried to defend the Ku Klux Klan”
From Law360 [$]: Why attorneys tend to use ChatGPT over legal-specific GenAI tools
From 404 Media [$]: “Cellebrite, the company which makes near ubiquitous phone hacking and forensics technology used by police officers around the world, has introduced artificial intelligence capabilities into its products, including summarizing chat logs or audio messages from seized mobile phones, according to an announcement from the company last month”
Legal AI Hallucinations
Iovino v. Michael Stapleton Associates [LawSites summary | Docket]
Bunce v. Visual Technology Innovations, Inc. et al [Order | Docket]
Merz v. City of Kalama (of note - the error is not fictitious case cites but a self-represented litigant who claimed that AI told them a point of civil procedure that was wrong) [Order | Docket]
Miscellaneous
The California Supreme Court, which oversees the State Bar, has issued a statement regarding last week’s bar exam, directing the agency “to plan on administering the July 2025 California Bar Examination in the traditional in-person format”
From LawSites: “KPMG Becomes First of Big Four To Practice Law in U.S., As Arizona Approves Its ABS License”
From Reuters: “Utah is scaling back a four-year-old program that loosened rules for delivering legal services in the state, leading nearly 30 businesses and law firms to exit or be eliminated, according to Utah Supreme Court orders and a court official”
Long Reads
Schwarcz, Daniel and Manning, Sam and Barry, Patrick James and Cleveland, David R. and Prescott, J.J. and Rich, Beverly, AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice (March 02, 2025). Available at SSRN: https://ssrn.com/abstract=5162111 [Summary]
Harris, David Evan Harris and Shull, Aaron, Generative AI, Democracy and Human Rights, Policy Brief No. 12, Centre for International Governance Innovation, Feb. 28, 2025, https://www.cigionline.org/publications/generative-ai-democracy-and-human-rights/
Melumad, Shiri and Yun, Jin Ho, Experimental Evidence of the Effects of Large Language Models versus Web Search on Depth of Learning (January 20, 2025). The Wharton School Research Paper, Available at SSRN: https://ssrn.com/abstract=5104064 or http://dx.doi.org/10.2139/ssrn.5104064
Murray, Michael D., Artificial Intelligence for Learning the Law: Generative AI for Academic Support in Law Schools and Universities - Report of Experiments (September 04, 2024). Available at SSRN: https://ssrn.com/abstract=4946680 or http://dx.doi.org/10.2139/ssrn.4946680