Evaluating the SQE

The SQE changes the way people become solicitors. Our objectives for the SQE are:

• to provide greater assurance of consistent high standards at point of admission
• to encourage the development of new and diverse pathways to qualification, which are responsive to the changing legal services market and are responsive to the changing legal services market and promote a diverse profession by removing artificial and unjustifiable barriers.

The SQE replaces the old system of qualification, in which multiple providers decided who should qualify as a solicitor, with a single rigorous assessment. You can find detailed information about the assessment on the SQE website.

We have also made significant changes to the requirements for solicitor training – we require aspiring solicitors to have:

• A degree or equivalent experience, such as an apprenticeship. Under the old system, we only accepted a qualifying law degree or a non-law degree plus a law conversion course.
• Two years' qualifying work experience (QWE), which can be completed in up to four organisations that provide legal services, whether or not we regulate them. Under the old system, aspiring solicitors had to get a training contract - a two-year period of recognised training in an organisation we regulated. (This requirement does not apply to qualified lawyers who become a solicitor through the SQE route).

We also require all aspiring solicitors to meet our character and suitability requirements.

Why we are evaluating the SQE

We will evaluate the SQE so that we can understand the extent to which it achieves its objectives, maximise the benefits of our reforms and understand any unintended impacts.

This means our evaluation needs to assess, among other things:

• the fairness, validity and reliability of the SQE assessment
• the impacts of our reforms on the education and training of solicitors, and on the legal market, including the equality and diversity impacts and perceptions of the change among different groups
• whether any unintended consequences of our reforms emerge over time.
• what wider factors, which may be partly or fully outside our control, have a positive or negative impact on the objectives of the SQE.

Our evaluation needs to take place over a period of ten years so that we can understand any longer-term effects because, for example:

• it will take time for significant numbers of aspiring solicitors to qualify through the SQE and start providing legal services to the public
• education and training providers, and legal services providers, might take time to adapt to our reforms.

In the interests of transparency and accountability, we are committed to publishing information about the evaluation of our reforms and engaging key groups in that process.

Pye Tait’s evaluation framework for the SQE comprises the first of five phases of our evaluation programme.

The framework enables us to apply a systematic and structured approach to assessing the impact of our reforms. In line with Pye Tait’s recommendation, we will revisit the framework on a regular basis.

Focusing on change

Our evaluation of the SQE focuses on the changes we want to see as a result of our reforms so that the objectives of the SQE are met.

Some of these changes are within our control and should happen quickly. For example, we will be able to assess the fairness, validity and reliability of the SQE as soon as we start to get data from the assessment.

Other changes might take longer to happen because they are influenced by a range of factors, which might be outside our control. For example, the increased availability of flexible and cost-effective routes to qualification depends on how the education and training market responds to our reforms over time.

Indicators

Pye Tait also created a set of indicators, which specify what data and information we need to measure to understand the impacts of our reforms.

We already have arrangements in place to measure some indicators. For example, we will collect assessment data, which is an indicator of the fairness, validity and reliability of the SQE. This will include data to help us consider equality, diversity and inclusion implications.

Research will be needed to help us measure other indicators. For example, research to understand candidates’ experiences of finding training, which is an indicator of the expanded availability of training routes.

You can find a summary of the indicators in Tables 2, 3 and 4 of Pye Tait’s report.

There are five phases of our evaluation programme, with equality, diversity and inclusion implications considered throughout.

We will share more information about each phase of our evaluation programme in due course. Phases four and five start at least five years after the introduction of the SQE and will be shaped by the findings of the earlier phases.

2021: Phase one

The evaluation framework developed by Pye Tait completes phase one.

2022: Phase two

Published in March 2023, this phase consisted of an initial perception study of both the SQE assessment and qualifying work experience.

These initial studies give us an indication of what is and is not perceived to be working well, which we can monitor over time. They will act as a helpful baseline for future surveys.

2024: Phase three

We plan to complete this phase three years after the introduction of the SQE, and it will additionally include an initial evaluation of the market impacts of our reforms, including any equalities impacts.

By updating the initial perception study from 2022 we can identify, for example:

• if any unintended consequences of our reforms have emerged since the initial perception study
• if any barriers to the successful delivery of our objectives, that were identified in the initial perception study, have been overcome.

We plan to explore the impacts of the QWE reforms in more depth, which might include looking at:

• the quality of QWE
• whether our QWE reforms have started to address any of the current barriers to qualifying as a solicitor
• lessons learnt by training providers and other stakeholders

By 2024, the wider impacts of our reforms on the education and training market for solicitors should start to emerge. We will design the approach and topics of our initial evaluation of these impacts closer to the time. We expect it to include an analysis of the information we have already collected about our reforms as a whole.

2026 – 2028: Phase four

We plan to complete this phase from year five to year seven after the introduction of the SQE, and it will consist of:

• a further perception study
• a further study about the impacts of QWE
• research about the impacts of our reforms on the market and the experiences of candidates from every background
• an independent evaluative study of the SQE assessment.

To identify the longer term impacts of our reforms, the perception study and the study about the impacts of QWE will build on the findings from phases two and three.

Research about the impact of our reforms on the market and the experiences of candidates might analyse, for instance:

• data, including diversity data, about the profile of people entering the profession since our reforms were introduced, and how it compares to the baseline position
• information about candidates’ experiences of qualifying as solicitors since our reforms were introduced, and how they compare to experiences under the old system.

The independent evaluative study of the SQE assessment will be completed by an independent assessment expert or experts. They will have information about the SQE assessment since it was introduced, as well as the work we have done to evaluate it. To maintain the independence of the study, the framework and methodology that will be used will be determined by the expert(s).

2031: Phase five

The final phase of our evaluation programme will take place ten years after the SQE was introduced.

Our plans for this phase will be shaped by the findings of earlier phases. Pye Tait has recommended repeating and updating the elements of phase four, because it will take this long (and possibly longer in some cases) for the impacts of our reforms to be fully realised.

Introduction

The new Solicitors Qualifying Examination (SQE) is key to achieving our corporate objective of setting and maintaining high standards for the profession. It is also an important step towards the development of new and diverse pathways to qualification and the promotion of a diverse profession.

The SQE is a single, rigorous assessment for all aspiring solicitors. The first of the two part assessment is SQE1, which tests the application of legal knowledge through two 180 question exams looking at 13 areas of law. The second is SQE2 which tests 16 legal skills assessments across different practice areas. Ethics and professional conduct are tested throughout and SQE1 must be passed before a candidate can take SQE2.

Our assessment provider for the SQE is Kaplan. This means they are also the approved registered provider for the end point assessment for solicitor apprentices. We have been appointed as the quality assurer by the Institute for Apprentices and Technical Education. Find out more about the solicitor apprentice standard and assessment plan.

Our quality assurance (QA) framework sets out how we will safeguard the standard and assure the quality of the SQE.

Quality assurance outcomes

This framework aims to achieve these outcomes (or alert us to risks) by creating a bank of evidence showing that:

• the assessments are valid (including realistic) and reliable
• the Threshold Standard is being consistently applied
• assessments are fair and free from bias
• decisions about candidate performance are fair and consistent
• the assessment questions are aligned to the assessment specifications and the assessment blueprint
• ethics and professional conduct pervade the SQE and candidate performance in ethics and professional conduct is to a standard appropriate to professional practice
• the assessment is secure
• the quality of the assessment, including delivery is to a good standard
• the information provided to candidates and training providers is clear, accessible, accurate and timely

Principles

The framework is built on a set of principles to make sure:

• the assessment is relevant to professional competence
• it has consistent and evidence-based decision making
• there are reliable results
• there is confidence in the assessment
• there is continuous improvement
• equality, diversity and inclusion has been appropriately considered throughout

These principles are:

• a culture of transparency and openness is fostered and maintained
• all assessments are valid, reliable and fair
• policies and processes are effective, efficient and applied in a consistent way
• effective mechanisms and procedures for identifying and managing risk are used
• best practice and continuous improvement are achieved

Our quality assurance model

We are using the three lines of defence model of quality assurance and risk management, to which we have added a fourth:

• First line – Kaplan’s documented policies, processes and controls, manage risk to the standard and quality of the SQE on a day to day-to-day basis
• Second line – Kaplan monitors and reviews the application of its policies and processes for compliance, consistency and risk in the development and delivery of the SQE
• Third line – we check that Kaplan has a robust framework in place to:
• quality assure key activities associated with the development and delivery of the SQE
• monitors and obtain evidence of compliance
• provides effective controls and management of risk
• Fourth line - an SQE Independent Reviewer monitors and reviews the development and delivery of the assessment by Kaplan and our own monitoring and quality assurance activity

The framework

Our QA framework reflects the steps we have taken in the SQE’s development and what is in place for its ongoing quality assurance:

• developed the assessment specifications (and assessment plan for solicitor apprentices), through extensive stakeholder engagement and piloting to make sure that the SQE is fit for purpose
• appointed Kaplan to develop and deliver the assessment following a rigorous tendering process
• contractual obligations which set out requirements and service levels in the development and delivery of the SQE, and agreed mechanisms for monitoring these
• working closely with Kaplan to review and agree all processes and policies to make sure they are ready to deliver the SQE
• a cycle of monitoring activity reflected in the third line of defence
• independent oversight by an external reviewer and psychometrician reflected in the fourth line of defence
• regular reporting on the quality of the SQE

Monitoring

Our quality assurance activities are focussed in the third line of defence and provide for a continuous cycle of monitoring including:

• checking processes and policies are consistently applied by Kaplan
• checking the assessments are valid, reliable and fair, and align to the assessment specifications and assessment blueprint
• assessment sampling and observations of live assessments by subject matter experts
• making sure the Threshold Standard is consistently applied
• checking decisions made relating to candidates are fair and consistent and according to agreed policies and practices
• data collection, analysis and evaluation of candidate performance
• checks for barriers to equality, diversity and inclusion
• reviewing policies and processes
• obtaining stakeholder feedback
• monthly meetings with Kaplan to check against service levels

Our quality assurance activities will take place before, during and after each assessment. This allows us to manage risk and to take action where needed.

External oversight

In addition to the checks and monitoring that we will carry out (including those made by our subject matter experts), we have a further level of assurance from two independent sources:

• the SQE Independent Reviewer who will provide oversight of the quality assurance of the SQE by Kaplan and us
• the independent psychometrician who will provide oversight and quality assurance of Kaplan’s SQE psychometric and assessment analysis

They will make sure Kaplan’s internal controls and quality assurance are effective, and risk management mechanisms put in place are robust and working effectively.

Reporting 

A report on candidate performance is published after each Assessment Board.

From 2023, we have published annual reports reviewing the SQE and candidate performance over several delivery points.

Continuous improvement and review

Our ongoing monitoring, and that built into Kaplan’s quality assurance framework, is designed to aid continuous improvement as follows:

• a regular cycle of policy and process review, including a review of this framework
• meetings following each assessment to capture issues arising from delivery and agree solutions for future assessment runs
• feedback from meetings with the SQE Oversight Group
• feedback from the Trailblazer Steering Group
• feedback from the SQE reference group to include membership from key stakeholder groups including:
• special interest and diversity groups
• universities and training providers
• Junior Lawyers Division
• The Law Society