Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada

Abstract

Background: Consistent healthcare decision making requires systematic consideration of decision criteria and evidence available to inform them. This can be tackled by combining multicriteria decision analysis (MCDA) and Health Technology Assessment (HTA). The objective of this study was to field-test a decision support framework (EVIDEM), explore its utility to a drug advisory committee and test its reliability over time.

Methods: Tramadol for chronic non-cancer pain was selected by the health plan as a case study relevant to their context. Based on extensive literature review, a by-criterion HTA report was developed to provide synthesized evidence for each criterion of the framework (14 criteria for the MCDA Core Model and 6 qualitative criteria for the Contextual Tool). During workshop sessions, committee members tested the framework in three steps by assigning: 1) weights to each criterion of the MCDA Core Model representing individual perspective; 2) scores for tramadol for each criterion of the MCDA Core Model using synthesized data; and 3) qualitative impacts of criteria of the Contextual Tool on the appraisal. Utility and reliability of the approach were explored through discussion, survey and test-retest. Agreement between test and retest data was analyzed by calculating intra-rater correlation coefficients (ICCs) for weights, scores and MCDA value estimates.

Results: The framework was found useful by the drug advisory committee in supporting systematic consideration of a broad range of criteria to promote a consistent approach to appraising healthcare interventions. Directly integrated in the framework as a "by-criterion" HTA report, synthesized evidence for each criterion facilitated its consideration, although this was sometimes limited by lack of relevant data. Test-retest analysis showed fair to good consistency of weights, scores and MCDA value estimates at the individual level (ICC ranging from 0.676 to 0.698), thus lending some support for the reliability of the approach. Overall, committee members endorsed the inclusion of most framework criteria and revealed important areas of discussion, clarification and adaptation of the framework to the needs of the committee.

Conclusions: By promoting systematic consideration of all decision criteria and the underlying evidence, the framework allows a consistent approach to appraising healthcare interventions. Further testing and validation are needed to advance MCDA approaches in healthcare decisionmaking.

Figure 1

Study plan.

Figure 2

Mean weights for decision criteria of the MCDA Core Model from the drug advisory committee. A five-point weighting scale was used with 1 for lowest weight and 5 for highest weight.

Figure 3

Mean scores for decision criteria of the MCDA Core Model for the appraisal of tramadol by the drug advisory committee. A four-point scoring scale was used with 0 for lowest score and 3 for highest score.

Figure 4

MCDA value estimate of tramadol for chronic non-cancer pain. Weights were normalized across the 14 criteria and scores are presented on a scale of 0 to 1. *MCDA value estimate was obtained using a linear model combining normalized weights and scores for each decision criterion. For an intervention to achieve close to 1 on this scale, it would have to cure an endemic disease, demonstrate major improvement in safety, efficacy, and patient-reported outcomes compared to limited existing approaches and result in major healthcare savings.

Figure 5

Impact of contextual criteria on tramadol appraisal by the drug advisory committee.