Document Type : Original Article

Authors

1 Center for Medical Decision Making, Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands

2 Dutch Institute for Clinical Auditing, Leiden, The Netherlands

3 Department of Surgery, Rhode Island Hospital, Warren Alpert Medical School of Brown University, Providence, RI, USA

4 Department of Research, Netherlands Comprehensive Cancer Organization (IKNL), Utrecht, The Netherlands

5 Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, The Netherlands

6 Department of Surgery, Erasmus MC Cancer Institute, Rotterdam, The Netherlands

7 # A full list of the collaborators of the NBCA Consortium is provided at the end of the article

Abstract

Background 
Quality indicators (QIs) are widely used to benchmark hospital performance and improve quality of care but are often based on expert opinion rather than data-driven assessment. This study aims to evaluate QIs, using a framework that assesses four criteria: Feasibility, discriminative ability, validity, and reliability.
 
Methods 
We used data from the Dutch breast cancer registry (NABON Breast Cancer Audit, NBCA) and included all surgically treated breast cancer patients in the Netherlands between 2021-2023. Eighteen QIs were evaluated. Feasibility was determined by QI numerator completeness, with >90% data availability considered feasible. Discriminative ability was assessed by between-hospital variation in QI scores, where an interquartile range (IQR) >10% indicated good discrimination. Validity was evaluated by the impact of case-mix adjustment and considered low when (pseudo-) R² <0.10. The (pseudo-)R² reflects the proportion of variance in QI scores explained by all case-mix variables in a regression model. Reliability was assessed by rankability, the proportion of between-hospital variation not due to chance and, therefore, explainable by quality of care. Rankability >75% was considered high. 
 
Results 
After exclusion of one QI with feasibility <1%, feasibility ranged from 80.2%-100%, and 15 QIs were feasible. Overall, the IQR ranged from 1 to 36, with 8 QIs having an IQR higher than 10, indicating good discriminative ability. The (pseudo-)R² ranged from 0.01-0.53, with 11 QIs showing low case-mix impact. Rankability ranged from 0-69%, with none of the QIs having a high rankability. None of the QIs met all preset criteria, but six QIs met at least three out of four criteria. 
 
Conclusion 
The QIs employed by the Dutch breast cancer registry fulfilled most criteria, but rankability is a concern and requires specific attention, especially for public reporting. Our results show the importance of considering feasibility, discriminative ability, validity, and reliability when evaluating QIs, and these should also be taken into account when developing new QIs.

Keywords