Observational studies of treatment effectiveness: worthwhile or worthless?

Abstract

Observational studies which evaluate effectiveness are often viewed with skepticism owing to the fact that patients are not randomized to treatment, meaning that results are more prone to bias. Therefore, randomized controlled trials remain the gold standard for evaluating treatment effectiveness. However, it is not always possible to conduct randomized trials. This may be due to financial constraints, for example, in identifying funding for a randomized trial for medicines that have already gained market authorization. There can also be challenges with recruitment, for example, of people with rare conditions or in hard-to-reach population subgroups. This is why observational studies are still needed. In this manuscript, we discuss how researchers can mitigate the risk of bias in the most common type of observational study design for evaluation of treatment effectiveness, the cohort study. We outline some key issues that warrant careful consideration at the outset when the question is being developed and the cohort study is being designed. We focus our discussion on the importance of deciding when to start follow-up in a study, choosing a comparator, managing confounding and measuring outcomes. We also illustrate the application of these considerations in a more detailed case study based on an examination of comparative effectiveness of two antidiabetic treatments using data collected during routine clinical practice.

Keywords: diabetes mellitus; effectiveness; epidemiology; public health; therapeutics.

Figure 1

Forest plot comparing our case study (Sharma et al) with meta-analyses of previous RCT examining change in HbA_1c (mmol/mol) between sitagliptin and sulfonylurea as add-on to metformin. Notes: Weights, where present, are from fixed-effects meta-analysis (Mantel–Haenszel method), although random-effects estimates (DerSimonian–Laird method) were identical. Source: Adapted from Sharma M, Beckley N, Nazareth I, Petersen I. Effectiveness of sitagliptin compared to sulfonylureas for type 2 diabetes mellitus inadequately controlled on metformin: a systematic review and meta-analysis. BMJ Open. 2017;7(10):e017260. Abbreviations: Dur, duration; Mean diff, mean difference; Sita, sitagliptin; Sulf, sulfonylureas; Tot, total participants; RCT, randomized controlled trial; Obs, observational study; NA, not applicable.