Desiderata for sharable computable biomedical knowledge for learning health systems

Abstract

In this commentary, we work out the specific desired functions required for sharing knowledge objects (based on statistical models) presumably to be used for clinical decision support derived from a learning health system, and, in so doing, discuss the implications for novel knowledge architectures. We will demonstrate how decision models, implemented as influence diagrams, satisfy the desiderata. The desiderata include locally validate discrimination, locally validate calibration, locally recalculate thresholds by incorporating local preferences, provide explanation, enable monitoring, enable debiasing, account for generalizability, account for semantic uncertainty, shall be findable, and others as necessary and proper. We demonstrate how formal decision models, especially when implemented as influence diagrams based on Bayesian networks, support both the knowledge artifact itself (the "primary decision") and the "meta-decision" of whether to deploy the knowledge artifact. We close with a research and development agenda to put this framework into place.

Keywords: Bayesian analysis; decision analysis; decision support; knowledge engineering; predictive modeling.

Figure 1

Asthma risk. A, Bayesian network (built in Netica™). Rectangles represent probabilistic nodes, the bar graphs, the probability distributions. The arrows between nodes reflect conditional  probabilistic relationships (numbers not shown). B, ROC resulting from the Bayesian network. Two models are reflected: one based on probability estimates elicited from experts and the other from data in the RMRS electronic health record system. The ROCs almost overlap each other. The AUC for each is 0.7015

Figure 2

Threshold model of decision making. The horizontal line reflects the probability of the outcome driving the decision. The vertical up arrow, what probability the decision maker holds before gaining information; the vertical down arrow, the probability after that information

Figure 3

Sensitivity analysis for the lead‐testing problem. Higher values of expected value are preferred to lower values, so the screen‐and‐test strategy (△) is preferred between the 2 blood lead risk thresholds of .0066 and .139 mcg/dL. To the left of that interval, the do‐nothing strategy (○) is preferred, and to the right, the test strategy (□) is preferred. In 1994, the general US probability was .26,22 to the right of the interval, so “test” was preferred (and indeed recommended). In the case of Flint, MI, the strategy of screen and test had the highest expected value before and after the water switch.23 The value of spending resources to get a better estimate of the probability of a lead value between 5 and 10 mcg/dL can is related to the distance between the best and second‐best strategies, wherever the locale's current estimate lies on the x‐axis

Figure 4

An algorithm depicting a decision rule for blood lead screening. If a child has any risk factors for lead poisoning shown, s/he should be tested with a blood lead level. A rule like this is easily implemented in all EHR‐based CDSSs

Figure 5

Example influence diagram for deciding whether to test a child, or children in general, for lead poisoning. At the point of the decision, whether to test blood lead, the decision maker knows the patient's risk factors (anemia, siblings with high lead level themselves, and whether their home was built before 1960). These risk factors have a probabilistic impact on the actual blood lead level, which is not observed, but whose probabilistic relationship (sensitivity, specificity) to blood lead test result is known. If the measured blood level is above a cutoff (not shown), then treatment occurs certainty (deterministically). The value (utility) to the patient will depend on what the actual blood lead level was, as well as the effectiveness of treatment; this value results from neurological sequelae and costs. The you SHOULDT components are labeled. (See supplement for details)

Figure 6

The knowledge artifact localization cycle

Figure 7

Bayesian meta‐model for debiasing. Specific biases come from Delgado‐Rodriguez56