SQE 1 performance reports lack technical detail, argues Professor Richard Moorhead — but regulator insists new exam will meet ‘high standards’
The incoming head of Exeter University Law School has criticised the pilot test of stage one of the new Solicitors Qualifying Examination (SQE), claiming it “obscures as much as it reveals” and did not meet “basic reporting standards”.
Professor Richard Moorhead, who joins the Russell Group university next month, claims the lack of “technical but reasonably robust and open analysis” in the pilot reports “suggests potentially significant issues about validity, fairness, and reliability”.
The pilot, run by Kaplan, the legal education outfit tasked with delivering the SQE, saw over 300 candidates sit a preliminary version of SQE1 earlier this year. The exam featured three multiple-choice papers of 120 questions, followed by a practical skills test consisting of one legal research and two legal writing exercises.
Responding to Kaplan’s feedback, the Solicitors Regulation Authority (SRA) revealed last week it was considering ditching the practical skills element of SQE1 after it failed to reach the “high standard of accuracy required”.
Writing in a blog post, Moorhead said:
“I’d have hoped for some examples of questions (some that candidates found easy, some that they found hard) and a detailed, open, if rather narrow, analysis of a statistical notion of reliability… The example questions with pass/fail rates would also be enormously helpful in generating understanding and perhaps acceptance of the tests.”
Addressing the apparent lack of information regarding those who sat the exam, Moorhead said: “We are told that those selected to sit the test are broadly representative but this is cold comfort.” The report, the academic continued, tells us “about representativeness rather than showing us the data to help us evaluate that claim”. This, he argued, is a “frankly bizarre way to go about dealing with data”.
Referencing the SRA’s goal of delivering a rigorous, “world class” assessment, Moorhead concluded:
“It is world class in the way that British volleyball is world class. If the SRA really took a decision based on these reports, I’d be genuinely shocked. There must be a more open, honest, technical report and they should publish it.”
Speaking last week, Paul Philip, SRA chief executive, said: “The SQE will help build trust that all qualifying solicitors are meeting consistent, high standards.”
The SQE, due to come into force in September 2021, will replace both the Legal Practice Course (LPC) and Graduate Diploma in Law (GDL). A pilot on SQE2 is due to run in December.
Update: 3:27pm
Julie Brannan, SRA director of education and training said: “We have discussed SQE design and development with stakeholders extensively since Kaplan was appointed and will continue to do so. We will also be publishing sample questions and an indicative pass mark on them, but it was never our intention to do that at this point.
“We are pleased to read that Richard accepts that MCQs can be more sophisticated than commonly accepted. As we said, we are pleased with how the SQE 1 pilot went. It provides confidence that the core part of SQE1 is appropriately rigorous, while helping us to improve it further.”