Performance of a Deep Neural Network Algorithm Based on a Small Medical Image Dataset: Incremental Impact of 3D-to-2D Reformation Combined with Novel Data Augmentation, Photometric Conversion, or Transfer Learning

Abstract

Collecting and curating large medical-image datasets for deep neural network (DNN) algorithm development is typically difficult and resource-intensive. While transfer learning (TL) decreases reliance on large data collections, current TL implementations are tailored to two-dimensional (2D) datasets, limiting applicability to volumetric imaging (e.g., computed tomography). Targeting performance enhancement of a DNN algorithm based on a small image dataset, we assessed incremental impact of 3D-to-2D projection methods, one supporting novel data augmentation (DA); photometric grayscale-to-color conversion (GCC); and/or TL on training of an algorithm from a small coronary computed tomography angiography (CCTA) dataset (200 examinations, 50% with atherosclerosis and 50% atherosclerosis-free) producing 245 diseased and 1127 normal coronary arteries/branches. Volumetric CCTA data was converted to a 2D format creating both an Aggregate Projection View (APV) and a Mosaic Projection View (MPV), supporting DA per vessel; both grayscale and color-mapped versions of each view were also obtained. Training was performed both without and with TL, and algorithm performance of all permutations was compared using area under the receiver operating characteristics curve. Without TL, APV performance was 0.74 and 0.87 on grayscale and color images, respectively, compared to 0.90 and 0.87 for MPV. With TL, APV performance was 0.78 and 0.88 on grayscale and color images, respectively, compared with 0.93 and 0.91 for MPV. In conclusion, TL enhances performance of a DNN algorithm from a small volumetric dataset after proposed 3D-to-2D reformatting, but additive gain is achieved with application of either GCC to APV or the proposed novel MPV technique for DA.

Keywords: Artificial intelligence; Coronary artery computed tomography angiography; Data augmentation; Deep neural network; Medical imaging; Photometric conversion; Transfer learning.

Fig. 1

GUI for segmentation of the coronary artery system. It includes capabilities for production of the following: (1) multiple orthogonal or oblique multi-planar reformatted or thin-maximum intensity projection 2D images (left sided 2 × 2 panel); (2) a stacked short-axis image series of a coronary artery [right edge strip], with manually applied tinting (red) reflecting local presence of atherosclerosis; and (3) centerline-dependent rotatable coronary artery 3D “branching tree” display (upper, between 2 × 2 panel and right edge strip), with artery enhancement (light-blue) indicating manual selection of artery-of-interest, and ball marker (dark-blue) and segment overlay (red) indicating specific level of manually demarcated atherosclerotic plaque

Fig. 2

Circumferentially arranged straightened-MPR displays of a diseased coronary artery/branch, all longitudinally co-registered to the shared centerline and then surface-illumination processed, produced a “stretched-appearing” volume image (top). Rotation of each volume image with creation of unique ray-traced (RT) projections every 10° (only 9 shown), produced multiple RT representations per artery/branch (bottom)

Fig. 3

DNN algorithm training with Inception-V3. The additional fully connected ReLU layer and Sigmoid output layer are added at the end of the DNN as shown (right). A sample input of DA coronary artery representations is also shown (left)

Fig. 4

To assess incremental impact of the following proposed methods on DNN algorithm performance: (1) DA (by MPV “mosaicking”), (2) photometric conversion (by GCC), and (3) TL (by initialization with pre-trained weights), eight models reflecting the possible combinations of aforementioned variables were developed; they included APV vs MPV in gray-scale (left), as well as APV vs MPV in color-map (right), both without and with application of TL

Fig. 5

Changing AUC based on applications of proposed DA, GCC, and/or TL methods is shown. The increase in performance from MPV-based DA (right) over APV use alone (left), as well as from the utilization of pre-trained model weights (TL with ImageNet) (bottom), compared to random-weight initialization of training (i.e., no TL) (top), demonstrate both the individual and additive value of the proposed novel DA and TL methods towards yield in DNN algorithm classification