Methodological expectations for demonstration of health product effectiveness by observational studies

Abstract

The issue of assessing the effectiveness of health technologies (drugs, devices, etc.) through observational studies is becoming increasingly important as registration and market access agencies consider them in their evaluation process. In this context, observational studies must be able to provide real demonstrations of a level of reliability comparable to those produced by the conventional randomized controlled trial (RCT) approach. The objective of the roundtable was to establish the acceptability criteria for an observational study (non-randomized, non-interventional study) to be able to provide these demonstrations, and possibly serve as a confirmatory study for registration and market access authorities, the construction of therapeutic strategies or the development of recommendations. In order to do this, the study must be a real confirmatory study respecting the hypothetical-deductive approach and guaranteeing the absence of HARKing and p-hacking by attesting to the establishment of a protocol and a statistical analysis plan, recorded before any inferential analysis. It must also be part of a formalized approach to causal inference and demonstrate that it correctly identifies the causal estimand sought. The study should ensure that there is no residual confusion bias by taking into account all confounding factors affecting the comparison, which should be determined by a formal approach (such as a graphical causality approach, DAGs). Residual confusion bias diagnoses by forgery and nullification analysis should be non-existent. The study shall be at low risk of bias, in particular selection bias, among others by using a target test emulation design. Overall type I error risk should be strictly controlled. The absence of selective publication of results and selection bias should be ensured.

Keywords: Critical reading; Decision-making; Health technology assessment; Market access; Observational studies; Real world data; Real world evidence; Regulatory agencies.