Identifying Existing Methods for Assessing Indirectness in Evidence Synthesis: A Scoping Review

Abstract

Introduction: Given the characteristics of the literature in the field of rehabilitation, which is marked by substantial clinical heterogeneity, poor reporting, and complexity in terms of interventions and outcomes, conducting meta-analyses can be complex or, in some cases, impossible. In particular, the concept of clinical heterogeneity is addressed within the indirectness domain of the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approach, which is assessed "by evaluating how closely the population, intervention, comparison, and outcomes in the available evidence align with those specified in the question of interest".

Objectives: As a first step toward developing a structured tool to assist systematic review authors in determining whether downgrading for indirectness is warranted, assessing the severity of indirectness, and translating these judgments into appropriate levels of certainty downgrading in rehabilitation evidence synthesis according to GRADE’s threshold approach, this study aims to identify existing structured methods (i.e., tools, checklists and scales) for assessing indirectness in any biomedical field.

Methods: This is a scoping review. MEDLINE, Embase, and CINAHL were searched using specifically designed search strategies up to May 20th, 2026. In addition, the GRADE Book, the Cochrane Handbook for Systematic Reviews of Interventions, the Joanna Briggs Institute Manual for Evidence Synthesis – 2024 Edition, and the Agency for Healthcare Research and Quality (AHRQ) website were consulted. Eligible studies were articles or book chapters describing the development, implementation, empirical validation, or use of any tool, checklist, or scale, aimed at assessing, measuring, or generally investigating indirectness (or related concepts) in evidence synthesis. Study selection and data extraction were performed by two independent authors.

Results: From 2593 individual records identified through database searching, two tools were identified, namely the Clinical Diversity In Meta-analyses (CDIM) tool and the Clinical Diversity in Pediatric Meta-analyses (CDIPM) tool. The latter is a modified version of the former (specifically designed for paediatric populations), available as a conference proceeding only. Both tools include four domains (i.e., setting, population, intervention, and outcome) and 11 items, each to be scored as low, moderate, or high clinical heterogeneity/diversity (0-2 points). Additionally, indirectness tables for both a single trial and the entire body of evidence, and a summary table for applicability were identified from the GRADE Book and the AHRQ Methods Guide for Comparative Effectiveness Reviews on applicability, respectively. The former included four subdomains (i.e., population, intervention, comparison and outcome) and four possible options regarding whether the evidence is sufficiently direct (i.e., Yes, Probably Yes, Probably No, and No), informing an overall judgement of indirectness across the subdomains. The latter requires a narrative assessment of the applicability of the evidence across five domains (Population, Intervention, Comparators, Outcomes, and Setting).

Conclusions: The findings highlight the paucity of structured methods available for assessing indirectness. At the same time, the identified methods largely share the same domains, likely suggesting that these may represent the key components that a tool for assessing indirectness should address. This work represents the first step that will inform the development of a tool to assess indirectness in the field of rehabilitation evidence synthesis.