A six-year repeated evaluation of computerized clinical decision support system user acceptability

Abstract

Objective: Long-term acceptability among computerized clinical decision support system (CDSS) users in pediatrics is unknown. We examine user acceptance patterns over six years of our continuous computerized CDSS integration and updates.

Materials and methods: Users of Child Health Improvement through Computer Automation (CHICA), a CDSS integrated into clinical workflows and used in several urban pediatric community clinics, completed annual surveys including 11 questions covering user acceptability. We compared responses across years within a single healthcare system and between two healthcare systems. We used logistic regression to assess the odds of a favorable response to each question by survey year, clinic role, part-time status, and frequency of CHICA use.

Results: Data came from 380 completed surveys between 2011 and 2016. Responses were significantly more favorable for all but one measure by 2016 (OR range 2.90-12.17, all p < 0.01). Increasing system maturity was associated with improved perceived function of CHICA (OR range 4.24-7.58, p < 0.03). User familiarity was positively associated with perceived CDSS function (OR range 3.44-8.17, p < 0.05) and usability (OR range 9.71-15.89, p < 0.01) opinions.

Conclusion: We present a long-term, repeated follow-up of user acceptability of a CDSS. Favorable opinions of the CDSS were more likely in frequent users, physicians and advanced practitioners, and full-time workers. CHICA acceptability increased as it matured and users become more familiar with it. System quality improvement, user support, and patience are important in achieving wide-ranging, sustainable acceptance of CDSS.

Keywords: Computerized decision support system; Decision support; Quality improvement; User acceptability; User satisfaction.

Figure 1

Timeline of relevant CHICA events and survey waves.

Figure 2. Results of user responses by question and year

Percents indicate portion agreeing, disagreeing, or responding neutrally to survey questions in each year. Neutral responses are shown here divided evenly along the agree and disagree axis. Negatively worded questions are reverse scored such that all results are shown with leftward responses being unfavorable to the CDSS and rightward responses being favorable. Numbers may not add to 100% due to rounding.

Figure 3. Adjusted logistic regression models of responding favorably acceptability survey

Each question represents a separate, adjusted model. Odds ratio estimates and 95% confidence intervals are depicted. Freq user = frequent CHICA user.

Figure 4. Summary of logistic regression models of responding favorably towards CHICA

Each point represents an odds ratio from a separate, adjusted model for each core acceptability question. The reference year is 2011. Transparent bars below point estimates show overlapping 95% confidence intervals. Asterisks denote significance at alpha = 0.05.