Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Aug 28;4(3):e12905.
doi: 10.2196/12905.

A Reinforcement Learning-Based Method for Management of Type 1 Diabetes: Exploratory Study

Mahsa Oroojeni Mohammad Javad  1 , Stephen Olusegun Agboola  2   3 , Kamal Jethwani  2 , Abe Zeid  4 , Sagar Kamarthi  4
Affiliations

A Reinforcement Learning-Based Method for Management of Type 1 Diabetes: Exploratory Study

Mahsa Oroojeni Mohammad Javad et al. JMIR Diabetes. .

Abstract

Background: Type 1 diabetes mellitus (T1DM) is characterized by chronic insulin deficiency and consequent hyperglycemia. Patients with T1DM require long-term exogenous insulin therapy to regulate blood glucose levels and prevent the long-term complications of the disease. Currently, there are no effective algorithms that consider the unique characteristics of T1DM patients to automatically recommend personalized insulin dosage levels.

Objective: The objective of this study was to develop and validate a general reinforcement learning (RL) framework for the personalized treatment of T1DM using clinical data.

Methods: This research presents a model-free data-driven RL algorithm, namely Q-learning, that recommends insulin doses to regulate the blood glucose level of a T1DM patient, considering his or her state defined by glycated hemoglobin (HbA1c) levels, body mass index, engagement in physical activity, and alcohol usage. In this approach, the RL agent identifies the different states of the patient by exploring the patient's responses when he or she is subjected to varying insulin doses. On the basis of the result of a treatment action at time step t, the RL agent receives a numeric reward, positive or negative. The reward is calculated as a function of the difference between the actual blood glucose level achieved in response to the insulin dose and the targeted HbA1c level. The RL agent was trained on 10 years of clinical data of patients treated at the Mass General Hospital.

Results: A total of 87 patients were included in the training set. The mean age of these patients was 53 years, 59% (51/87) were male, 86% (75/87) were white, and 47% (41/87) were married. The performance of the RL agent was evaluated on 60 test cases. RL agent-recommended insulin dosage interval includes the actual dose prescribed by the physician in 53 out of 60 cases (53/60, 88%).

Conclusions: This exploratory study demonstrates that an RL algorithm can be used to recommend personalized insulin doses to achieve adequate glycemic control in patients with T1DM. However, further investigation in a larger sample of patients is needed to confirm these findings.

Keywords: Q-learning; T1DM; diabetes treatment; insulin dose prescription; machine learning; reinforcement learning; type 1 diabetes mellitus.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
The agent-environment interaction in reinforcement learning.
Figure 2
Figure 2
Q-value update function.
Figure 3
Figure 3
Optimal policy function.
Figure 4
Figure 4
Reward function.
Figure 5
Figure 5
Random action selection function.
Figure 6
Figure 6
Error function.
See this image and copyright information in PMC

References

    1. American Diabetes Association. [2019-04-22]. Statistics About Diabetes: Overall Numbers, Diabetes and Prediabetes http://www.diabetes.org/diabetes-basics/statistics/
    1. Centers for Disease Control and Prevention. [2019-04-22]. CDC Newsroom https://www.cdc.gov/media/pressrel/2010/r101022.html.
    1. Martinez-Millana A, Jarones E, Fernandez-Llatas C, Hartvigsen G, Traver V. App features for type 1 diabetes support and patient empowerment: systematic literature review and benchmark comparison. JMIR Mhealth Uhealth. 2018 Nov 21;6(11):e12237. doi: 10.2196/12237. http://mhealth.jmir.org/2018/11/e12237/ - DOI - PMC - PubMed
    1. National Institute of Diabetes and Digestive and Kidney Diseases. [2019-04-22]. Kidney Disease Statistics for the United States http://www.niddk.nih.gov/healthinformation/healthtopics/kidneydisease/ki....
    1. The United States Renal Data System. [2019-04-22]. 2016 Annual Data Report https://www.usrds.org/adr.aspx.