Guiding the design of evaluations of innovations in health informatics: a framework and a case study of the SMArt SHARP evaluation

Abstract

Development of health information systems innovations is necessary to create a better future for health and health care, but evaluating them is challenging. This paper examines the problem of evaluating health IT projects in which innovation is agile, adaptive, and emergent, and in which innovation diffusion and production are interlinked. We introduce a typology of mindsets for evaluation design that are typically used in health informatics: optimality, contingency, and usefulness, and make the case for a modularity mindset. We propose a model that shifts the unit of analysis from an evaluation as a whole, to specific modules of an evaluation, such as purpose, target, and methods. We then use retrospective participant observation to illustrate the approach using a case study: the ONC SHARP Harvard project developing the SMArt platform (smartplaforms.org). We find that the proposed modular approach to evaluation design provides a balanced alternative to standard archetypical designs on the one hand, and fully custom-made designs, on the other hand.

Figure 1.

Overlapping realms of evaluation of health informatics innovation (a) Friedman and Wyatt, 1997 & . (b) Greenhalgh et al., 2004. (c) Greenhalgh et al., 2010b. (d) Patton, 2011

Figure 2.

Complexity profile: (a) pure customization, (b) pure standardization, and (c) hybrid. Adapted from Bar-Yam (1997)