But surprisingly good at drafting contacts
ChatGPT has been going from strength to strength recently but a new study may have finally found its limits.
Northern Irish firm SG Murphy Solicitors pitted the popular chatbot against high street lawyers in a number of legal scenarios and found that it struggled to compete.
One of the big problems they encountered was that 9% of the chatbot’s answers were factually incorrect. Though to be fair to the bot, the limitation list on its website includes the disclaimer: “may occasionally generate incorrect information”.
The report gives an example of one of these incorrect answers where the bot was asked: “What do I need to prove to claim a prescriptive easement in the UK?” In the bot’s response, it advised the imaginary client that they would need to prove necessity and that prescriptive easements were rare — both of which the authors said were incorrect.
There was also a lack of detail in three-quarters of the chatbot’s responses, meaning they were below the quality expected from their human counterparts. The report suggests the AI’s information was at a “basic and general level”, and its advice “does not match the quality level that we would expect to receive from an experienced lawyer”.
This lack of detail highlighted another key difference between the AI and a real lawyer: AI doesn’t ask questions. The researchers observed that, unlike real lawyers, ChatGPT made no requests for further information and which limited its ability to give tailored advice.
It was, however, able to provide information tailored to jurisdiction. When a question contained information that related to a particular jurisdiction, the bot picked up on this and adjusted its response accordingly. One example of this saw the bot successfully detailing the grounds for divorce in Northern Ireland.
Because of these drawbacks, the study’s authors came to the conclusion that while ChatGPT has a great deal of potential to help lawyers do their job more efficiently and effectively, in the short-term it cannot compete with experienced real-life lawyers due to its inability to provide tailored, precise, and — most importantly — correct advice.
However, they did highlight the tool’s aptitude for drafting simple agreements, indicating that this is the application for which it would be most useful.
Legal Cheek has reported many stories concerning the AI tool recently. It has been used by a Colombian judge to assist in a legal ruling, asked to write poetry, and even successfully answered questions on the Watson Glaser test.
