Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'

Abstract

Objectives: To describe the development of the Claim Evaluation Tools, a set of flexible items to measure people's ability to assess claims about treatment effects.

Setting: Methodologists and members of the community (including children) in Uganda, Rwanda, Kenya, Norway, the UK and Australia.

Participants: In the iterative development of the items, we used purposeful sampling of people with training in research methodology, such as teachers of evidence-based medicine, as well as patients and members of the public from low-income and high-income countries. Development consisted of 4 processes: (1) determining the scope of the Claim Evaluation Tools and development of items; (2) expert item review and feedback (n=63); (3) cognitive interviews with children and adult end-users (n=109); and (4) piloting and administrative tests (n=956).

Results: The Claim Evaluation Tools database currently includes a battery of multiple-choice items. Each item begins with a scenario which is intended to be relevant across contexts, and which can be used for children (from age 10 and above), adult members of the public and health professionals. People with expertise in research methods judged the items to have face validity, and end-users judged them relevant and acceptable in their settings. In response to feedback from methodologists and end-users, we simplified some text, explained terms where needed, and redesigned formats and instructions.

Conclusions: The Claim Evaluation Tools database is a flexible resource from which researchers, teachers and others can design measurement instruments to meet their own requirements. These evaluation tools are being managed and made freely available for non-commercial use (on request) through Testing Treatments interactive (testingtreatments.org).

Trial registration numbers: PACTR201606001679337 and PACTR201606001676150; Pre-results.

Keywords: evidence based medicine; hared decision making; health literacy; multiple-choice; outcome measurement; patient education.

Figure 1

Overview and timeline of the development process.

Figure 2

Example of formats.

Figure 3

Examples of incorrectly completed multiple-choice questions.

Figure 4

Distribution of correct answers in pilots.