Ahead of a broader Centre for Progressive Policy series on smarter public spending in the face of severe long term fiscal constraints, this blog explores the challenges of measuring performance in the public sector. It argues that the UK has been a pioneer in this field but that more needs to be done to make sense of issues like investing in prevention. The blog calls for more nuanced, mixed method approaches to measuring public sector productivity, so that we can better understand whether public spending is actually working to achieve the things we care about like improved health and educational outcomes.

At a time when the Chancellor, Jeremy Hunt talks about a drive to boost public sector productivity, there are real concerns about sector cuts and public finance sustainability. In this election year, one of the most central questions being asked of whoever takes power, is how they can do more with less. The outcome to this question impacts not just livelihoods but lives.

The UK, and many other developed countries, are being confronted with significant demographic challenges arising from an aging population and rising dependency ratios, that being the number of dependents in the total working age population. These factors will impact the demand for public services and the ability to provide them. Addressing these changes requires either increased resources or enhanced productivity within the public sector. In the words of the Institute for Government, “If [public service] productivity were to increase by 1%, public services could provide the same service using fewer staff and in effect save £2.4bn.”[1] This highlights the importance of productivity growth to public service provision and why this concept is so central to ensuring that not only we use scarce resources more efficiently, but also at the same time improve the quality of services for everyone.

For such an important indicator, it is crucial to ensure that our measurement processes reflect the reality. This is something that has long been recognised in the policy space and the UK has made pioneering contributions to this field, for instance, through the Atkinson Review of 2005 which recommend best practice methods and approaches that could be used to measure UK government output. However, there are new, emerging challenges which prompted the government this year to undertake a review of the way we measure public service productivity. Why is measuring productivity in government services difficult? What has been done about it so far and what more could be done? Why are we concerned about this issue, especially in areas such as health? What are some of the emerging challenges and how can we possibly deal with them?

Opening up Pandora’s Box

There are two aspects to productivity measurement in public services: measuring inputs and measuring outputs. As far as the former is concerned, the biggest concern relates to measuring the nature of complexity.[2] Numerous public goods and services exhibit a considerable level of complexity, demanding various, often immeasurable, contributions from multiple individuals and sources. As an illustration, the UK government's Annual Report on Major Projects outlines public infrastructure initiatives that are notably intricate, high-risk, and/or innovative, requiring diverse areas of expertise and involving a multitude of stakeholders and providers. Even the creation of seemingly straightforward monitoring or budget reports may involve the participation of several public officials, and gauging their exact contributions proves challenging.[3]

This problem has worsened in the face of digitalisation, where most digital services are characterised by even more complex production functions and supply chains.

As far as measuring outputs in public services is concerned, the exercise is rendered more difficult for four reasons.

First, governments usually provide services such as education, health, policing and so on – and output measurement in services is much more complicated than in the case of goods, even if one were to look at the private sector.

Secondly, most government services lack market transactions, and where they occur, subsidies heavily distort them.[4] Lack of a proper price mechanism means that determining the economic value of public services becomes difficult, as consumers cannot express how much they think a service is worth through quantities purchased or prices paid, making it more difficult to elicit an output value.

Thirdly, a big chunk of government services is composed of collective goods (such as defence, police) that are not consumed individually.

Lastly, the quality of public services evolves over time, and failing to account for these changes can result in inaccurate assessments of output value. For instance, specific healthcare procedures have notably improved in terms of safety and effectiveness over the years. Output measures that overlook these quality improvements will underestimate the actual value of the produced output. However, quantifying quality in numerical terms is inherently challenging.[5]

Shifting Landscape

Until the 1990s, public service productivity was assessed based on the notion that output equalled input costs, implying constant productivity. Economist Tony Atkinson sought to address this limitation, leading to the influential Atkinson Review (2005), revolutionising public service productivity measurement not just in the UK but globally. The review emphasised the importance of measuring the quality of public output, not just the quantity. This means analysing how much value is added in treating patients or teaching pupils, rather than just how many patients are treated or how many pupils are taught. This is done using a cost-weighted output index (CWAI).[6],[7]

However, like any other method, this too suffers from several limitations. It can only provide aggregate information – at the level of an organisation, sector, or the whole public service. This is certainly important but doesn’t offer the complete picture. There usually is substantial variation within and across public sector institutions and so characterising the productivity of complete sectors or systems using one metric masks the substantial differences in performance and management. This makes it difficult to identify best practices and good reforms.[8] Moreover, by capturing detailed measures of performance and productivity at the micro-level, it becomes easier to identify the relationships and potential elements of public-sector productivity.[9] This facilitates targeted policy interventions.

It is important we develop approaches that analyse productivity at several different levels – micro, meso and macro – as each would offer us unique insights into areas for improvement. The UK has made some commendable effort over the past few years in incorporating several micro-level approaches, however, more work in this direction is certainly required.[10]

The Health Productivity Conundrum

Why talk about productivity in health? As Centre for Progressive Policy’s report Funding Fair Growth has highlighted, owing to demographic and cost pressures, public spending in health must rise by 50 billion by the end of 2030, just to maintain things the way they are currently. In such a scenario, it is important to make sure that we are pumping money into a productive system. But this is easier said than done. Apart from the challenges in the measurement of public service productivity, services such as health are also mired with a plethora of other issues.

Between 2004 and 2017, NHS productivity increased two and a half times in comparison to the rest of the UK economy. On the other hand, NHS productivity witnessed a sharp decline of 25.6% from 2019-20 to 2020-21. There is evidence to argue that this decrease can be attributed not only to significant disruptions in elective and emergency care operations due to COVID-19 but also to limitations in the scope of productivity measurement.[11] Given the productivity goals that are expected of the NHS, it is absolutely important that our health productivity numbers reflect the reality.

More doesn’t mean better

There are some obvious flaws with the existing system. The current framework of health productivity measurement centres around healthcare activity, with the assumption that increasing both the volume and quality of healthcare will result in greater overall health—the ultimate objective of any health system. Another common assumption that underpins our current measurement framework is that more expensive forms of care inherently contribute more to improving health than their less costly counterparts.[12] However, this may not be the case. It is possible that there are many cost-effective procedures that lead to better results and on the other hand, many costly procedures may only improve health marginally.

Measuring prevention

In the face of rising health challenges, governments around the world are increasingly adopting a more preventive approach to health. However, existing healthcare productivity metrics are inadequate for gauging the enduring health advantages of public health policies. There often is a time lag between investment in such policies and the actual production of outcomes, whereas our measures are focussed on year-to-year production of outputs.[13] As our system measures productivity based on the amount of output produced, it is also possible that with greater focus on prevention, we end up underestimating NHS productivity. If preventive policies reduce the need for people to go to the hospital and reduce the amount of treatments conducted, our measures would suggest that productivity has fallen based on healthcare activity. It is easy to see why this is problematic.

Outputs ≠ Outcomes

There is another major shortcoming of our current measurement system which stems from equating outputs to outcomes. This is because better health outputs in the short-term need not necessarily lead to a healthier population in the future. This has long been recognized in the policy space but there are practical concerns around measuring outcomes.

Firstly, it takes time for outputs to transform into outcomes and as discussed before, this requires long-term measures of productivity which is difficult from a statistical perspective.

Secondly, outcome measures are sometimes easier to manipulate and far harder to reach consensus on.

As an example, during the period of UK Labour governments from 1997 to 2010, there was significant emphasis placed on educational outcomes such as exam performances of school leavers, with the assertion that increasing pass rates reflected improved educational quality. Nevertheless, numerous critics contended that the perceived improvements were a result of grade inflation.[14] On the other hand, it is also possible that outcomes are not entirely driven by outputs. Take the healthcare sector, for example; enhancements in surgical success rates and recovery times might be linked to the community's overall improvement in fitness and health, a factor unrelated to the quality of healthcare output.[15]

Conclusion

It is not hard to see why productivity measurement in public services a walk in the park isn’t. There is no one ‘right’ method or model in this case, as each approach suffers from both strengths and limitations. Such a situation calls for integral and multilevel measurement. This involves measuring an indicator in several different ways and/or at several different levels. For instance, apart from traditional measurement of output in terms of say, the volume of heart attack treatments, we could measure outcome in this case as a weighted index of the Quality Adjusted Life Years (QALY) gained from heart attack treatments which is a better reflection of health outcomes.[16] Another example would be to supplement traditional staff productivity statistics with information and insights from hospital management surveys – the former would tell us how productive workers are whereas the latter could help us discover what worker characteristics and management practices lead to better performance. A similar approach for assessing private sector productivity led to breakthrough findings in the field of productivity analysis.[17]

We can complement macro-level approaches with micro-level approaches, short-run indices with long-run indices, and output-based measures with outcome-based measures.[18] This is known as ‘methodological triangulation’, which simply refers to the practice of using more than one kind of method to study a phenomenon. The importance of such an approach in measuring public service productivity has, time and again, been highlighted by researchers.[19], [20] This might not give us a complete picture, but we’ll surely end up with a better one than we have today. It is encouraging to see the Office of National Statistics (ONS) take a methodologically diverse approach to public service productivity in its latest economic ‘nowcasting’ experimental statistics.

In our upcoming work, the Centre for Progressive Policy will be exploring how we can improve the current state of productivity and efficiency in public services. Yet, an integral aspect of this journey involves acknowledging and rectifying any misconceptions we may have held, which is only possible when we have a more comprehensive, complete and truer picture of productivity in our public services. In this never-ending quest for improvement, then, it is crucial that we leverage the power of what is known as multimethodology.