ASSESSING THE RISK OF BIAS, INDIRECTNESS, AND THE CERTAINTY IN THE BODY OF EVIDENCE ADDRESSING PEOPLE’S VALUES, UTILITIES, OR IMPORTANCE OF OUTCOMES

Abstract

To make informed healthcare decisions, it is essential to consider not only how much an intervention works but also how important the affected health outcomes are. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach is widely used to evaluate the certainty of evidence in systematic reviews, health technology assessment, and guidelines. GRADE provides a transparent and structured framework that considers multiple domains, including risk of bias (ROB), indirectness, inconsistency, imprecision, publication bias, and upgrading factors. However, the initial GRADE guidance on assessing the certainty of evidence in the importance of outcomes or values and preferences could not offer detailed operationalization for the evaluation of ROB or indirectness at the level of individual studies, because valid tools were not available. Valid and reliable tools are critical for such assessments, as they directly inform the overall certainty ratings. Recognizing this gap, we developed and validated two novel tools: the ROBVALU tool for assessing risk of bias and the DIRECTVALU tool for assessing indirectness in individual studies addressing the importance of outcomes or utility values. This work emphasizes the need for rigorous individual study assessments to strengthen the overall certainty of evidence using the GRADE approach. We followed a sequential mixed-methods approach starting with a qualitative approach to develop ROBVALU and DIRECTVALU tools, followed by a quantitative phase involving user testing and vi psychometric testing to ensure reliability and validity. This was followed by a three-step modified Delphi process with experts for final refinement of the tools. This work includes an update of the GRADE guidance for assessing certainty of the evidence in people’s values, utilities, or the importance of health outcomes. These tools and the updated GRADE guidance support a more structured and transparent approach to make judgments across the overall body of this type of evidence. Ultimately, this will lead to improvements in the GRADE certainty of evidence assessments in systematic reviews, health technology assessments, and health guidelines, that would ensure the trustworthiness of our recommendations that supports healthcare decision-making.