Quality indicators and weight

Traditionally, the SPO framework is generally adopted for constructing an indicator system, which preferably covers the contents of resources needed, service process and health benefits that patients preserve through receiving services.37–39 In this study, the SPO was used as the primary indicator for categorising quality indicators of cancer care. Regarding indicator weight, outcomes indicators had the largest weight, while structure indicators had the lowest weight in this study. Outcome indicators, with a weight of over a half, reflected the experts’ substantial emphasis on patients’ health benefits from received care. Up to date, the grade certification of cancer hospitals in China mainly assesses the resources that they own and service processes, which is different from experts’ perspectives on quality cancer care. From an international perspective, outcome indicators have an important position in the quality evaluation of healthcare as well.40 For structure and process domains, there were more indicators included because of their easiness to monitor and access in China. In addition, based on the boundary value criteria, a core set of indicators was reserved, which could be used when resource is limited or collecting too many indicators is impracticable.

Outcome indicators covered five aspects, as the secondary indicators showed, namely patient safety indicators, treatment outcome indicators associated with survival, improved health status and recovery, patient satisfaction indicators, and efficiency and cost indicators. This design of secondary indicators for cancer care outcome basically aligned with the six dimensions IOM proposed in 2001.22 Patient safety mainly involved mortality and adverse events. Patient satisfaction with health providers’ services is an emerging source of hospital evaluation, where information is collected from patients’ perspectives and it embodies the principle of patient-centredness.30 Efficiency and cost indicators reflect the service’s accessibility and affordability to some extent. The extremely vital indicators in cancer care evaluation are survival, improved health status and recovery, where a longer survival time and a quicker recovery period are the ultimate goals of care.41.

Process indicators cover the entire streamline of diagnosis, treatment and rehabilitation. Experts place more emphasis on the diagnostic accuracy and conformity of treatment guidelines. In particular, multidisciplinary diagnosis and treatment has been recognised as a priority for patients with cancer, which has been widely referred to and adopted both worldwide and in China.29 42 43 In addition, care comfort and symptom alleviation are recognised as necessities, which represent patient-centred principles through showing concerns and supporting considerable care for patients’ pain, sleep disturbance, malnutrition and depression. Structure indicators involve four aspects, namely staff, information system, service capability, and research and training. A superior information system could help simplify workflow and enhance service efficiency. The use of intelligent medical decision support systems, such as treatment checks and reminders, drug monitoring, and clinical path recommendation, may help decrease the occurrence of medical errors.

Overall, this set of quality indicators developed in China had same framework and similar measuring aspects with the international quality systems. A umbrella review summarised hospital performance indicators globally.44 This review found over half of studies covered efficient, effective, patient-centred and safe aspects, and some studies covered responsive governance, staff orientation and timeliness. The set of quality indicators developed in this study is lacked of indicators related with responsive governance, which refers to that hospitals respond to community needs and ensure the continuity and coordination of services. In addition, compared with quality indicators developed in OECI BENCH-CAN project, this study had little consideration in organisational governance and leadership.23 This may indicate the overlook for governance among hospital administrators and researchers in China. The optimal way to measure governance and leadership in China is currently lacking. For tertiary indicators, diversity existed compared with international measures. Indicator of adverse events reporting system was adopted in this study, while some studies developed tools to identify and monitor patient adverse events from patient’s or staff’s perspectives, such as CTCAE and PRO-CTCAE.45 46 These novel ways remain to be tested in China in the future.

Indicator feasibility and practical application

Some experts expressed concerns about the feasibility of two indicators (survival rate and improved health status and function recovery) in their comments during the modified Delphi process. These experts also claimed that survival rate was important for cancer care evaluation, but its utilisation should be cautious for the following reasons. First, many hospitals cannot calculate this indicator due to the lack of long-term follow-up, especially for those in less developed areas. Second, patients may receive treatments at various hospitals; thus, the survival rate does not necessarily reflect the effects of care from only one hospital. When survival rate is used for hospital quality evaluation, patients receiving treatments at various hospitals should not be enrolled. Third, if there are too few patients in one hospital, the estimation of the survival rate is not robust and may cause misleading evaluation. An expert discussing whether survival statistics make sense for every hospital expressed similar concerns.47 International experts also emphasised that outcome indicators are vulnerable to inadequate sample size and statistical power.41 48 For these reasons, the survival rate is claimed to be optional when applying this set of quality indicators. Regarding improved health status and function recovery indicator, different measuring scales across hospitals and measuring frequency at least two times bring challenges for their data collection and application. In addition, the variability of electronic medical reporting systems at various regions of China will impact the practical implementation of quality indicators. According to the National Health Commission statistics, 232 of 2817 tertiary hospitals were rated 5 and over in the grades system of electronic medical records(grades ranged 1–8).

Future work and challenges

Constructing a set of indicators for measuring the quality of care in cancer hospitals is an essential step for quality evaluation and improvement; however, future work needs to bridge the gap between tool construction and practical application. The validation and practical application for this tool are conducting at about 30 tertiary cancer hospitals across China. The data acquisition and preparatory data analyses are ongoing. Some important points are as follows:

Empirical validation of tools

Empirical studies should be conducted to test the validity and reliability of quality measures. Although every indicator of quality of care is identified based on previous studies, it remains to be verified whether a composite index after indicator scoring and weighting could appropriately discriminate a hospital’s care quality.49 50

Risk adjustment for indicators

The indicator set developed in our study helps to measure the quality of care and conduct comparisons across hospitals or across different periods. When conducting a comparison, risk adjustment is a prerequisite, as otherwise underestimation of performance for these practices serving vulnerable patients is possible.51 52 Previous studies have also found that risk adjustment played a larger role for outcome indicators than for process indicators, indicating the importance of casemix adjustment for outcome indicators.53

Public reporting of results

Publicly reported hospital performance enhances transparency, spurs quality improvement and empowers patient choice.54 For policy-makers and hospital governors, quality measuring and reporting could help target primary improvement points. Patients also have increasing enthusiasm for public reporting of hospital performance so that they can seek proper and better care.55 Ensuring that reports are clear, precise and understandable is also a critical step for the profound impact of hospital evaluation. League tables and funnel plots are applied to this end.56

Strengths and limitations

This study has some strengths. First, this study adapted scientific framework and method, and took both international evidence-based indicators and features of China’s health system into consideration. Second, the panel of experts comprised people from different regions across China, represented various expert backgrounds and had a high response rate. Third, this study provided new insights into the systemic measurement of cancer care from Chinese experts’ perspectives.

This study also has some limitations. First, the entire application of this tool is limited in other countries because of different health systems and social contexts. Second, some indicators with compromised feasibility were reserved because of their significance, but their practical application in some developing regions in China may be adopted partly or optimised accordingly. Third, patients’ viewpoints are not collected, therefore, it is lacking that how important these quality indicators are from a patient perspective. In addition, although this study tried to cover various regions across China, most experts included were from the Eastern area of China.