Taming Performance Variability of Healthcare Data Service Frameworks with Proactive and Coarse-Grained Memory Cleaning

Abstract

This article explores the performance optimizations of an embedded database memory management system to ensure high responsiveness of real-time healthcare data frameworks. SQLite is a popular embedded database engine extensively used in medical and healthcare data storage systems. However, SQLite is essentially built around lightweight applications in mobile devices, and it significantly deteriorates when a large transaction is issued such as high resolution medical images or massive health dataset, which is unlikely to occur in embedded systems but is quite common in other systems. Such transactions do not fit in the in-memory buffer of SQLite, and SQLite enforces memory reclamation as they are processed. The problem is that the current SQLite buffer management scheme does not effectively manage these cases, and the naïve reclamation scheme used significantly increases the user-perceived latency. Motivated by this limitation, this paper identifies the causes of high latency during processing of a large transaction, and overcomes the limitation via proactive and coarse-grained memory cleaning in SQLite.The proposed memory reclamation scheme was implemented in SQLite 3.29, and measurement studies with a prototype implementation demonstrated that the SQLite operation latency decreases by 13% on an average and up to 17.3% with our memory reclamation scheme as compared to that of the original version.

Keywords: data framework for applied health data; database; health data management; medical systems; memory reclamation.

Figure 1

Applications for SQLite.

Figure 2

Overall architecture of SQLite.

Figure 3

Memory reclamation scheme of SQLite.

Figure 4

SQLite large transaction processing. The subfigure at the bottom shows the transaction commit timeline. The gray box with R denotes the memory reclamation time duration and the black box with T indicates the data writing time duration of the transaction into the buffer.

Figure 5

Latency of 4 GB storage write varying the write size.

Figure 6

Memory reclamation scheme with proactive and coarse-grained cleaning.

Figure 7

Throughput of 10 KB transaction in four different versions of SQLite.

Figure 8

Throughput of 1 MB transaction in four different versions of SQLite.

Figure 9

SQLite Performance varying the threshold value.