Developing a Framework for Assessing the Applicability of the Target Condition in Diagnostic Research

Eve Tomlinson; Jude Holmes; Anne W.S. Rutjes; Clare Davenport; Mariska M. Leeflang; Bada  Yang; Sue Mallett; Penny Whiting

doi:10.54103/2282-0930/29258

Authors

Eve Tomlinson Population Health Sciences, Bristol Medical School, University of Bristol
Jude Holmes Centre for Medical Imaging, University College London
Anne W.S. Rutjes UniCamillus - Saint Camillus International University of Health and Medical Sciences
Clare Davenport Department of Applied Health Research, University of Birmingham ; NIHR Birmingham Biomedical Research Centre, University Hospitals Birmingham NHS Foundation Trust and University of Birmingham, Birmingham, UK
Mariska M. Leeflang Amsterdam University Medical Centers, University of Amsterdam
Bada Yang Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University
Sue Mallett Centre for Medical Imaging, University College London
Penny Whiting Population Health Sciences, Bristol Medical School, University of Bristol

DOI:

https://doi.org/10.54103/2282-0930/29258

Abstract

Introduction: Assessment of the applicability of primary studies to the review questions is an essential but often challenging aspect of systematic reviews of diagnostic test accuracy studies (DTA reviews).

Objectives: To explore review authors’ applicability assessments for the QUADAS-2 reference standard domain within Cochrane DTA reviews, by highlighting applicability concerns, identifying potential issues with assessments. Using the findings to develop a framework for assessing the applicability of the target condition as defined by the reference standard.

Methods: Methodological review. DTA reviews in the Cochrane Library that used QUADAS-2 and judged applicability for the reference standard domain as “high concern” for at least one study, were eligible. One reviewer extracted the rationale for the “high concern”, this was checked by a second reviewer. Two reviewers categorized the rationale inductively into themes, a third reviewer verified these. Discussions in the QUADAS development group (steering group and expert panel) regarding the extracted information informed framework development.

Results: We identified 50 eligible reviews. We identified potential issues over half (n=28) of the included reviews. In 7, applicability assessments deviated from QUADAS-2 definitions and in 21 applicability concerns were insufficiently described. Five themes emerged in the remainder: study uses different reference standard threshold to define the target condition (6 reviews), misclassification by the reference standard in the study such that the target condition in the study does not match the review question (11 reviews), reference standard could not be applied to all participants resulting in a different target condition (5 reviews), misunderstanding QUADAS-2 applicability (7 reviews), and insufficient information (21 reviews).

The Figure shows the framework that was informed by our findings and discussions. Our framework outlines four potential applicability concerns for the assessment of the target condition as defined by the reference standard: different sub-categories of the target condition, different threshold used to define the target condition, reference standard not applied to full study group, and misclassification of the target condition by the reference standard (Figure).

Conclusion: Clear sources of applicability concerns are identifiable, but several Cochrane review authors struggle to adequately identify and report them. We have developed the first applicability framework to guide review authors in their assessment of applicability concerns for the QUADAS reference standard domain. The development of a framework for the QUADAS-2 domains Patient selection and Index Test are ongoing.