The Tougher the Better: The Effect of GP Performance Thresholds in Scotland

Yan Feng

Yan Feng

General Practitioners (GPs) in the NHS are paid based on a mix of factors, including a pay-for-performance element, the Quality and Outcomes Framework (QOF), which was introduced in 2004.  QOF rewards GPs based on performance on specific indicators. Although participation in the QOF is voluntary, it represents a key source of potential income for GPs and nearly all participate. The QOF has been criticised, however, for having produced little or no improvement in GP performance; some studies have claimed that the QOF achievement has had a negligible impact on health outcomes.

The OHE’s Yan Feng and her colleagues take a closer look at the effect of changes in the QOF in Scotland in April 2006. They state that “the principal aim of this paper is to evaluate the effect of an increase in the maximum performance threshold in the QOF scheme on the performance of GPs in Scotland.”  The paper also examines whether GPs responded to the incentives differently according to the level of their performance before the change.

Under the QOF, GP practices are paid according to a linear schedule between a lower and upper threshold, which varies across indicators. The indicator set was revised in 2006; new indicators were introduced, some were retired and others revised.  Moreover, minimum performance thresholds were raised from 25% to 40% for all indicators. Maximum thresholds were raised for nine clinical indicators and left unchanged for 25 other indicators for which definitions also were unchanged.  This selective change provides a unique opportunity to analyse the response of GPs by comparing the performance on indicators with an increased threshold to performance on indicators that remained the same.

“The changes in the payment threshold under the QOF scheme may have differential effects on different groups of GPs”, the authors note. The research examines whether GP practices that had initially higher and initially lower performance responded differently.  The financial incentives for low-performing GPs were strongest: unless their performance improved, the amount of potential income foregone would be even greater than before the change.  High-performing GPs would have the least incentive to change.  The study also looks at a middle group.

Data used in the analysis were provided by the Scottish government’s Information Services Division and included financial years 2005/6 and 2006/7.  In total, the data include 40,704 observations across 24 indicators.

“The principal finding of this paper,” the authors state, “is that the increased maximum performance threshold under the QOF scheme in 2006 was differentially effective in improving GPs’ performance. Overall, the effect of the policy change in 2006 improved the performance of GPs in Scotland.” The research also bears out the hypothesis that incentives produced greater change among lower-performing GPs.

Download: Feng, Y., Ma, A., Farrar, S. and Sutton, M. (2012) The tougher the better: The effect of an increased performance threshold on the performance of general practitioners. Research Paper 12/02. London: Office of Health Economics.

New: Assessing the Performance of the EQ-VAS in the NHS PROMS Programme

Assessing the Performance of the EQ-VAS in the NHS PROMs ProgrammeThe NHS Patient Reported Outcome Measures (PROMs) programme, introduced in April 2009, is a significant development in the routine collection and use of patient reported outcome data.  Currently, data are collected from patients both before and after surgery for four elective surgical procedures in the NHS, with plans to expand the practice.  (The procedures are hip and knee replacement, groin hernia repair and varicose vein repair.)

Results from the PROMs data collected already are being used in a broad range of decision making contexts[1].  For example, comparisons of the changes in patient health before and after surgery are being used as one indicator of hospital performance[2].  Commissioners also are using the findings in evaluating the effectiveness and cost effectiveness of a range of services.  Ensuring that the data are as accurate and reliable as possible, then, is crucial.

The EQ-5D, used to collect the PROMs data, has two parts. The first, the EQ-5D profile, asks patients to classify their health based on self-assessed levels of problems (“no”, “some”,” extreme”).  The second is the EQ-VAS, which asks patients to indicate their overall health on a vertical visual analogue scale, ranging from “worst possible” to “best possible” health.  Historically, the EQ-VAS was a warm-up exercise for VAS valuations of EQ-5D, but it has evolved into an integral part of the measure.  This paper focuses on two concerns about the EQ-VAS: (1) difficulties with the data because patients may fail to respond to it as the instructions require and (2) substantial differences between patient responses on the EQ-VAS versus the EQ-5D profile and condition specific instruments.  According to the authors, these issues raise fundamental questions about the role and use of EQ-VAS in the EQ-5D instrument.

In particular, the paper analyses:

  1. The different ways in which patients complete the EQ-VAS and how their characteristics affect this
  2. How the different ways of completing the EQ-VAS currently are handled in coding the data and the effect this may have on data interpretation and analysis
  3. The relationship between the EQ-VAS, the EQ-5D profile and other summary score data in the NHS PROMs programme

Analyses to address the first two points — potential issues with patient responses and how to handle them – are based on matched before-and-after anonymised EQ-VAS responses from 200 patients across all four elective procedures.  The data included background characteristics, namely age, sex, and type of surgery.  The third set of analyses, comparing EQ-VAS data to index weighted measures, were based on patient-level NHS PROMs programme data linked to Hospital Episode Statistics.  The 331,951 anonymised patient records covered all four elective procedures from 1 April 2009 through 28 February 2011.

The authors’ findings suggest ways for improving EQ-VAS by improving both data collection and coding procedures.  They note, for example, that although 95% of patients completed the EQ-VAS in an unambiguous way, fewer than 50% completed it in the way that the instructions intend.  Moreover, the guidance provided by the EuroQol Group on coding the imperfect responses is insufficient.  The authors note that this potentially can result in unnecessary data wastage or variations across users in interpreting and coding that, in turn, may make data less comparable.  “All of these issues,” they note, “could be addressed by providing improved guidance on coding EQ-VAS data or revisiting the instructions for the EQ-VAS”.

The third set of analyses presented in the paper addressed whether and how patient reported outcomes vary using EQ-VAS versus EQ-5D or condition specific instruments.  The authors note that “Concern had emerged from the NHS PROMs programme that the EQ-VAS was not adequately reflecting the health gain for patients resulting from surgery, and was therefore a less useful and appropriate measure of health change than the EQ-5D profile or condition specific instruments.”  Their findings suggest that this is not the case; the relationship between the EQ-VAS and the EQ-5D profile are predictable and consistent. Moreover, some of the differences between the two are attributable to the characteristics of the particular weightings within the EQ-5D index.

Nevertheless, the results of analyses do confirm the observation of PROMs reports that there are clear differences between the EQ-VAS and index weighted EQ-5D and condition specific profiles.  Presumably, patients are freer to consider all components of “health” using the VAS than they are when restricted to “tick boxes” with the EQ-5D profile.  “In essence,” the authors state, ”the EQ-VAS is measuring a broader underlying construct than the EQ-5D profile or the condition specific instruments. This does not mean that the data it produces are less meaningful or useful. Indeed, in applications where the patients’ view of their overall health is the measurement goal, the EQ-VAS is prima facie more appropriate than the use of EQ-5D profile data weighted by general public preferences.”

In closing, the authors note that no discoverable research has been done on how patients or members of the general public interpret the upper and lower endpoints of the EQ-VAS – “best” and “worst” possible health.  Given the importance of the EQ-VAS, they urge that a better understanding be developed of differences in interpretation, the bases for these, and whether they may change with expectations, health or social circumstances.

Download Feng, Y., Parkin, D. and Devlin, N.J. (2012) Assessing the performance of the EQ-VAS in the NHS PROMs Programme. Research Paper 12/01. London: Office of Health Economics.



[1] See Devlin, N.J. and Appley, J. (2010) Getting the most out of PROMs: Putting health outcomes at the heart of NHS decision making. London: Office of Health Economics.

[2] See our recent blog on new research on analysing hospital variation with EQ-5D. 

New: Operationalising Value Based Pricing

Operationalising Value Based PricingRecent reforms to the National Health Service (NHS) in England include important changes in the regulation of prices for new medicines. From January 2014, the existing Pharmaceutical Pricing Regulation Scheme (PPRS) will be replaced by “value based pricing” (VBP) for branded medicines sold to the NHS. This will apply only to new medicines; those marketed before 2014 will continue to be governed by the PPRS.

To date, the UK Government has not been specific about how VBP will be implemented, but has indicated that calculations of value will extend beyond the QALY approach currently used by NICE.  The Department of Health has stated that it will include “the range of factors through which medicines deliver benefits for patients and society”.

In this OHE Research Paper, the authors identify and describe the full set of possible means by which value based pricing might be operationalised, categorise these by developing a taxonomy of approaches, and provide an initial assessment of the challenges, pros and cons of each.  They review the elements of value that could be considered and how these might be measured and valued,  combined into an overall assessment of a medicine’s value, and then linked to the maximum price the health service is willing to reimburse.  Finally, the means by which VBP is currently operationalised in a selection of countries is examined; both these and proposals for the UK are placed in the context of the taxonomy.

Download Sussex, J., Towse, A. and Devlin, N. (2011)  Operationalising Value Based Pricing of Medicines: A Taxonomy of Approaches. OHE Research Paper. London: Office of Health Economics.

QALYs v. Other Criteria in NHS Decisions

Just released is an OHE Research Paper that examines whether and how decisions about allocating resources for health care differ across the NHS and, particularly, in comparison to decisions by NICE.  If important differences exist, the efficiency of resources allocation may be affected negatively.

The appraisal of health care technologies by the UK’s National Institute for Health and Clinical Excellence (NICE) focuses on cost effectiveness, usually measured in terms of incremental cost per quality adjusted life year (QALY) gained.  According to the authors of this new OHE Research Paper, however, cost effectiveness often does not appear to be the dominant consideration in decisions about resource allocation made elsewhere in the NHS.

In this paper, the authors examine what factors affect NHS policy decisions by analysing 51 Impact Assessments (IAs) completed by the UK Department of Health (DH) in 2008-9. Reported by the DH for all new legislation and policy implementation, IAs are intended to help ‘develop better policy by careful consideration of the impact of relevant options on all those affected.’   The official guidance for IAs suggests that QALYs, when used, should be valued in money terms, but does not specify a particular method.

The main source of data for this research is the DH Publications website, from which were downloaded IA documents published in 2008-9.  Analysis showed that of the 51 IAs available at the time of research, only eight used QALYs to evaluate benefits, including four of those associated with the largest monetised benefits.  Ninety-three unique benefits other than QALY gains were identified as having been used to justify NHS policy decisions.  Most could be grouped into the following categories: improvement in health outcomes (cited in 26 of the 51 IAs), improvements in health service cost and efficiency (19 IAs), improvements in quality (15 IAs), and enhancing the patient and carer experience (11 IAs).  For 21 IAs, no monetary impact was estimated as the IA stated that the benefits of the chosen policy could not be monetised or quantified.

Given the use in most IAs of several criteria, not all monetised, the authors suggest that multiple criteria decision analysis (MCDA) may be useful in ensuring consistent use of criteria.  A recent OHE publication explains MCDA in greater detail.

Setting aside the issue of what cost and benefit perspective is ‘appropriate’, the authors point out that this research suggests that the perspectives adopted by NICE and the DH when assessing the impact of NHS spending are very different, and that this misalignment of aims has ‘clear implications’ for efficiency across the health care system.  A greater consensus about what NHS goals are, and the relative importance of those goals, could improve welfare, they conclude.

Download Shah, K.K., Praet, C., Devlin, N.J., Sussex, J.M., Appleby, J. and Parkin, D. (2011) Is the aim of the health care system to maximise QALYs? An investigation of ‘what else matters’ in the NHS. OHE Research Paper. London: Office of Health Economics.

New: Antimicrobial Drug Incentives

OHE has just made available Research Paper 11/02, Incentives for New Antimicrobial Drugs.

Antimicrobial resistance (AMR) is becoming a major global public health threat and has begun to command attention from European and US policy makers. An initial focus on monitoring AMR and conserving existing treatments by cutting down on misuse has been complemented by moves towards addressing the paucity of new drugs in the R&D pipeline of the pharmaceutical industry.

The authors identify five economic challenges: the utilisation externality; the lack of incentives for R&D arising from use restrictions, low prices, and scientific and regulatory challenges; the global joint sunk nature of R&D cost; the need for access to drugs in middle and low income countries; and failures in the market for point of care diagnostics.  Recommended is a hybrid combination of “push” and “pull” incentives: higher prices linked to targeted use with diagnostic tests and/or an AMC-based “prize” for registering (but not necessarily using) desired new drugs, linked also to push measures.

The authors see US and European collaboration on incentives as desirable, but not if achieving agreement leads to delays. Action on conservation needs to be global and linked to use of new products.  TRIPS provisions and national sensitivities on this issue present serious challenges, as seen in the 2010 reaction to evidence on the origins of NDM-1.

Download Towse, A. and Sharma, P. (2011) Incentives for R&D for new antimicrobial dugs. OHE Research Paper 11/02.  London: Office of Health Economics.

Valuing the National Clinical Assessment Service

The UK appears to be unique in its National Clinical Assessment Service (NCAS), intended specifically to help health care providers in the NHS deal with problem clinicians. Established in 2001, the NCAS provides support, on request, to all NHS organisations employing or contracting with doctors, pharmacists, and dentists. Currently, the NCAS is financed from central NHS funds, not from charges to NHS organisations. By 2013, however, the NCAS is to become self-funding (a decision made after completion of the research summarised below).

OHE’s latest Research Paper reports on a project meant to establish how much referrers in the NHS value NCAS’s services and the relative value placed on different attributes and types of service. Funded by an unrestricted grant from the NCAS, the project used a mix of qualitative and quantitative methods to elicit information from potential referrers, including a discrete choice experiment (DCE) that provided estimates of relative values and willingness to pay.

The qualitative aspects of the study included a literature review, semi-structured interviews with six senior NCAS staff members and with the medical, dental and pharmacy professions (BMA, BDA and RPS), and discussion groups with a total of 23 senior medical and other NHS managers in each of the four UK nations. The quantitative DCE elicited responses from 450 senior NHS managers responsible for managing clinical performance concerns that could be referred to the NCAS.

The DCE results suggest that the senior NHS managers who participated in the research valued a full package of support at £161.56 per year for each whole-time-equivalent doctor, dentist or pharmacist. Approximately 200,000 such health professionals currently are employed by or under contract to the NHS, making the value of a full package of services covering them all over £32 million per year. Currently, the NCAS provides all these services at an annual cost of £9 million (2008/09).

The paper discusses in some detail the nuances that were revealed by the research — for example, the determinants of variations in preferences for particular packages and in willingness to pay – and factors that may affect changes in demand for NCAS’s services over time.

Download Watson, V., Sussex, J., Tetteh, E. and Ryan, M. (2011) Managing poorly performing clinicians: The value of independent help. OHE Research Paper 11/01. London: Office of Health Economics.

Limits to Using QALYs in Cancer

Martina Garau

A quality-adjusted life year, or QALY, is a year of life adjusted for its quality or its value for the individual.  A year in perfect health would be equal to 1 QALY; the value for less than perfect health would decline as health declines.  QALYs are used to assess the value of medical interventions and may be used in decisions about allocating

health care resources.  In the UK for example, NICE uses QALYs to ‘compare different drugs and measure their clinical effectiveness’[1]; NICE recommendations determine whether health care interventions are covered under the NHS.  Clearly, then, ensuring that QALYs accurately reflect reality is crucial.

Koonal Shah, a health economist at the OHE, has been part of a team[2] researching how well QALYs assess cancer patients’ health gains from treatment.  He presented key findings in June 2010 at the annual conference of Health Technology Assessment International.

QALYs are calculated by multiplying the duration of each health state experienced by patients by the value ascribed to that state.  Health states are measured using instruments such as the EQ-5D; valuations are based on the preferences of a sample of the general population.

According to Mr Shah, existing methods for constructing QALYs may be deficient for cancer patients in three respects: descriptions of health state, valuation of health state and the source of values upon which measures are based.

Existing measures of health are either not sensitive enough or not attuned to cancer patients’ actual preferences.  For example, evidence suggests that the EQ-5D instrument does not capture the small changes in health that often are very important to cancer patients.

Valuation of health states for cancer patients using traditional methods also encounters problems.  For example, the time trade-off (TTO) method assumes that the rate at which people are willing to trade life expectancy for  improvements in quality of life is the same under all circumstances.  Research has shown, however, that (1) severely ill patients often are willing to sacrifice more life expectancy for smaller gains in health-related quality of life and (2) patients with less than one year to live are often unwilling to trade any time.

Whose values form the basis of health state valuations also matters.  NICE recommends that valuations of EQ-5D health states be based on the preferences of the general public. Descriptions of health states using EQ-5D, however, are too general to provide the public with a complete depiction of the patient experience. The public also tends to focus on the negative aspects of ill health, missing the fact that some areas of life are not seriously affected.  Finally, a discrepancy exists between the way patients actually adapt to ill health and healthy individuals’ perceptions of their own abilities to adapt. Overall, using the general public to value health states may overvalue interventions aimed at achieving perfect health and undervalue those aimed at prolonging life or achieving small improvements in HRQL.

Mr Shah proposes three possible ways forward, given these concerns.  First, improve descriptions by adding dimensions or levels to EQ-5D and/or develop cancer-specific instruments.  Second, develop the TTO approach so that it captures the nuances of the relationship between remaining life and quality of life.  And, third, make health state descriptions more comprehensive and/or realistic to achieve better-informed valuations.

Download Garau, M., Shah, K.K., Mason, A.R., Wang, Q., Towse, A. and Drummond, M.F. (2010) Using QALYs in cancer:  Review of the methodological limitations. Research Paper. London: Office of Health Economics.

See also: Garau, M., Shah, K.K., Mason, A.R., Wang, Q., Towse, A. and Drummond, M.F. (2011) Using QALYs in cancer:  A review of the methodological limitations.  Pharmacoeconomics. 29(8), 673-685.

 


[1]http://www.nice.org.uk/newsroom/features/measuringeffectivenessandcosteffectivenesstheqaly.jsp

[2] The team also includes colleagues at OHE and Anne Masson and Professor Mike Drummond of the University of York’s Centre for Health Economics.