Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Sep;40(9):883-899.
doi: 10.1007/s40273-022-01167-1. Epub 2022 Jul 15.

Best-Worst Scaling and the Prioritization of Objects in Health: A Systematic Review

Ilene L Hollin  1 , Jonathan Paskett  2 , Anne L R Schuster  2 , Norah L Crossnohere  2 , John F P Bridges  3
Affiliations

Best-Worst Scaling and the Prioritization of Objects in Health: A Systematic Review

Ilene L Hollin et al. Pharmacoeconomics. 2022 Sep.

Abstract

Background and objective: Best-worst scaling is a theory-driven method that can be used to prioritize objects in health. We sought to characterize all studies of best-worst scaling to prioritize objects in health, to assess trends of using best-worst scaling in prioritization over time, and to assess the relationship between a legacy measure of quality (PREFS) and a novel assessment of subjective quality and policy relevance.

Methods: A systematic review identified studies published through to the end of 2021 that applied best-worst scaling to study priorities in health (PROSPERO CRD42020209745), updating a prior review published in 2016. The PubMed, EBSCOhost, Embase, Scopus, APA PsychInfo, Web of Science, and Google Scholar databases were used and were supplemented by a hand search. Data describing the application, development, design, administration/analysis, quality, and policy relevance were summarized and we tested for trends by comparing articles before and after 1 January, 2017. Multivariate statistics were then used to assess the relationships between PREFS, subjective quality, policy relevance, and other possible indicators.

Results: From a total of 2826 unique papers identified, 165 best-worst scaling studies were included in this review. Applications of best-worst scaling to study priorities in health have continued to grow (p < 0.01) and are now used in all regions of the world, most often to study the priorities of patients/consumers (67%). Several key trends can be observed over time: increased use of pretesting (p < 0.05); increased use of online administration (p < 0.01), and decreased use of paper self-administered surveys (p = 0.02); increased use of heterogeneity analysis (p = 0.02); an increase in having a clearly stated purpose (p < 0.01); and a decrease in comparing respondents to non-respondents (p = 0.01). The average sample size has more than doubled, from 228 to 472 respondents, but formal sample size justifications remain low (5.3%) and unchanged over time (p = 0.68). While the average PREFS score remained unchanged at 3.1/5, both subjective quality and policy relevance trended up, but changes were not statistically significant (p = 0.06 and p = 0.13). Most of the variation in subjective quality was driven by PREFS (R2 = 0.42), but it was also positively assosciated with policy relevance, heterogeneity analysis, and using a balanced incomplete block design, and was negatively associated with not using developmental methods and an increasing sample size.

Conclusions: Using best-worst scaling to prioritize objects is now commonly used around the world to assess the priorities of patients and other stakeholders in health. Best practices are clearly emerging for best-worst scaling. Although legacy measures (PREFS) to measure study quality are reasonable, there may need to be new tools to assess both study quality and policy relevance.

PubMed Disclaimer

Conflict of interest statement

Ilene L. Hollin, Jonathan Paskett, Anne L.R. Schuster, Norah L. Crossnohere, and John F.P. Bridges have no conflicts of interest that are directly relevant to the content of this article.

Figures

Fig. 1
Fig. 1
Flow chart. BWS best–worst scaling
Fig. 2
Fig. 2
Perspective of best–worst scaling studies over time
Fig. 3
Fig. 3
Number of objects included in best–worst scaling studies
Fig. 4
Fig. 4
Number of objects per task included in best–worst scaling studies
See this image and copyright information in PMC

References

    1. Bridges JF, Selck FW, Gray GE, McIntyre JA, Martinson NA. Condom avoidance and determinants of demand for male circumcision in Johannesburg, South Africa. Health Policy Plan. 2011;26(4):298–306. doi: 10.1093/heapol/czq064. - DOI - PubMed
    1. Bridges JF, Gallego G, Kudo M, et al. Identifying and prioritizing strategies for comprehensive liver cancer control in Asia. BMC Health Serv Res. 2011;11(1):1–12. doi: 10.1186/1472-6963-11-1. - DOI - PMC - PubMed
    1. Peay H, Hollin I, Bridges J. Prioritizing parental worry associated with Duchenne muscular dystrophy using best–worst scaling. J Genet Counsel. 2016;25(2):305–313. doi: 10.1007/s10897-015-9872-2. - DOI - PubMed
    1. Tsai JH, Scheimann AO, McCandless SE, Strong TV, Bridges JFP. Caregiver priorities for endpoints to evaluate treatments for Prader-Willi syndrome: a best–worst scaling. J Med Econ. 2018;21(12):1230–1237. doi: 10.1080/13696998.2018.1528980. - DOI - PubMed
    1. Oakes AH, Garmo VS, Bone LR, Longo DR, Segal JB, Bridges JFP. Identifying and prioritizing the barriers and facilitators to the self-management of type 2 diabetes mellitus: a community-centered approach. Patient. 2017;10(6):773–783. doi: 10.1007/s40271-017-0248-6. - DOI - PubMed

Publication types

LinkOut - more resources