A Deep Learning Quality Control Loop of the Extrusion-based Bioprinting Process

Abstract

Extrusion-based bioprinting (EBB) represents one of the most used deposition technologies in the field of bioprinting, thanks to key advantages such as the easy-to-use hardware and the wide variety of materials that can be successfully printed. In recent years, research efforts have been focused on implementing a quality control loop for EBB, which can reduce the trial-and-error process necessary to optimize the printing parameters for a specific ink, standardize the results of a print across multiple laboratories, and so accelerate the translation of extrusion bioprinted products to more impactful clinical applications. Due to its capacity to acquire relevant features from a training dataset and generalize to unseen data, machine learning (ML) is currently being studied in literature as a relevant enabling technology for quality control in EBB. In this context, we propose a robust, deep learning-based control loop to automatically optimize the printing parameters and monitor the printing process online. We collected a comprehensive dataset of EBB prints by recording the process with a high-resolution webcam. To model multiple printing scenarios, each video represents a combination of multiple parameters, including printing set-up (either mechanical or pneumatic extrusion), material color, layer height, and infill density. After pre-processing, the collected dataset was used to thoroughly train and evaluate an ad hoc defined convolutional neural network by controlling over-fitting and optimizing the prediction time of the network. Finally, the ML model was used in a control loop to: (i) monitor the printing process and detect if a print with an error could be stopped before completion to save material and time and (ii) automatically optimize the printing parameters by combining the ML model with a previously published mathematical model of the EBB process. Altogether, we demonstrated for the first time how ML techniques can be used to automatize the EBB process, paving the way for a total quality control loop of the printing process.

Keywords: Automatic parameter optimization; Convolutional neuronal network; Extrusion-based bioprinting; Quality control.

Figure 1

(A) The main steps of the frame pre-processing pipeline, alongside example images of the output. (B) Examples of the data augmentation transformations applied during model training only. The transformations refer to the pre-processed frame in (A). Note that a composition of all these augmentations was applied to each frame in the training dataset.

Figure 2

(A) General overview of the CNN architecture, given by the repletion of convolutional blocks (“conv block” for short) for a number of times equal to the depth parameter. Note that each “conv block” reduces the input dimension (specified between brackets) by a factor of 2. (B) Different types of “conv blocks” tested during the model optimization procedure, namely, “simple,” “vgg,” and “resnet.”

Figure 3

(A) The flow diagram of the automatic parameter optimization algorithm. (B) An example of a 4-step optimization procedure. The two steps, namely, the “init calibration” and the “small perturbations” steps, are highlighted, alongside example printability windows used to calibrate the EM parameter. The dots in the figure correspond to the EM used for each step at the corresponding LH_i. Green dots represent a print that was predicted as “ok” by the DL model, while red dots one that was predicted with an error (“under_e” or “over_e” classes).

Figure 4

(A) The validation accuracy and loss for the selection experiments (presented as a mean across the 5-fold of the cross-validation procedure), as well as the number of parameters for each tested model. (B) The training loss and accuracy curves for the selected model (depth = 6 and a simple “conv block”). The dashed lines represent the original data points, while the solid lines are the results of a moving average filter (window size of 3) for better visualization. The red vertical dashed lines represent the epoch at which the model was saved during early stopping (epoch = 6). (C) The confusion matrix obtained by classifying the dataset using the model trained in (b). (D and E) The results of the classification invariance to zoom and focus and the grad-CAM activations on three example prints, respectively. The green border in (D) represents a correct prediction by the model (all images refer to an “ok” print).

Figure 5

Example plots of the prediction probability for each class over time. The vertical red dotted lines represent the print progress after which the model correctly classifies the print.

Figure 6

Example calibration procedure across four prints. For the case of LH_i = 0.7, from the initial printability window, we have that Δ = 0.31. The vertical red dotted lines represent the print progress after which the model correctly classifies the print.