Specific instructions for estimating unclearly reported blinding status in randomized trials were reliable and valid

Abstract

Objective: To test the reliability and validity of specific instructions to classify blinding, when unclearly reported in randomized trials, as "probably done" or "probably not done."

Study design and setting: We assessed blinding of patients, health care providers, data collectors, outcome adjudicators, and data analysts in 233 randomized trials in duplicate and independently using detailed instructions. The response options were "definitely yes," "probably yes," "probably no," and "definitely no." We contacted authors for data verification (46% response). For each of the five questions, we assessed reliability by calculating the agreement between the two reviewers and validity by calculating the agreement between reviewers' consensus and verified data.

Results: The percentage with unclear blinding status varied between 48.5% (patients) and 84.1% (data analysts). Reliability was moderate for blinding of outcome adjudicators (κ=0.52) and data analysts (κ=0.42) and substantial for blinding of patients (κ=0.71), providers (κ=0.68), and data collectors (κ=0.65). The raw agreement between the consensus record and the author-verified record varied from 84.1% (blinding of data analysts) to 100% (blinding of health care providers).

Conclusion: With the possible exception of blinding of data analysts, use of "probably yes" and "probably no" instead of "unclear" may enhance the assessment of blinding in trials.