But only just — bot performs as an average law student, Dr Ioannis Glinavos finds
A university academic has pitted ChatGPT against a first-year law degree exam paper in contract law, with underwhelming results.
Senior law lecturer at the University of Westminster, Dr Ioannis Glinavos, asked the bot to answer a selection of sample exam questions taken from a Pearson law textbook available online.
One scenario considered a contract to build a motor yacht for a computer tycoon and required ChatGPT to provide advice in response to supply chain issues and changes to a payment plan. Unfortunately, the bot’s response was marked as “tactical advice, not legal advice”, and unlikely to gain a pass mark in a law examination.
The bot’s responses continued to be hit and miss in all of the questions put to it, leading Glinavos to conclude its performance was equivalent to that of an average first-year law student. “The bot could sit in my class and I wouldn’t know the difference,” he said in the video (below).
Like most students the bot had areas of strength and weakness. Glinavos explained, “it can identify correctly the relevant legal issues but it struggles with analysis and authority.”
“The bot is very close to the average student performance, thus not easily detectable as an AI-produced answer,” said Glinavos. Ruling that ultimately the bot’s attempt at the exam paper scored around 45% but was still good enough to merit a pass.
He finishes with a warning that current AI-detection tools for exams are not yet advanced enough to competently detect students using these types of tools to cheat. “It will eventually lead to the abandonment of online timed exams that are not proctored,” he predicted.
