Accurate Medical Vial Identification Through Mixed Reality: A HoloLens 2 Implementation

Abstract

The accurate identification of medicine vials is crucial for emergency medical services, especially for vials that resemble one another but have different labels, volumes, and concentrations. This study introduces a method to detect vials in real-time using mixed reality technology through Microsoft HoloLens 2. The system is also equipped with an SQL server to manage barcode and vial information. We conducted a comparative analysis of the barcode detection capabilities of the HoloLens 2 camera and an external scanner. The HoloLens 2 effectively identified larger barcodes when they were 20-25 cm away in normal lighting conditions. However, it faced difficulties in detecting smaller barcodes that were consistently detected by the external scanner. The frame rate investigation revealed performance fluctuations: an average of 10.54 frames per second (fps) under standard lighting conditions, decreasing to 10.10 fps in low light and further reducing to 10.05 fps when faced with high barcode density. Resolution tests demonstrated that a screen resolution of 1920 × 1080 yielded the best level of accuracy, with a precision rate of 98%. On the other hand, a resolution of 1280 × 720 achieved a good balance between accuracy 93% and speed. The HoloLens 2 demonstrates satisfactory performance under ideal circumstances; however, enhancements in detecting algorithms and camera resolution are required to accommodate diverse surroundings. This approach seeks to help paramedics make quick and accurate decisions during critical situations and tackle common obstacles such as reliance on networks and human mistakes. Our new approach of a hybrid method that integrates an external Bluetooth scanner with the MR device gives optimal results compared to the scanner-only approach.

Keywords: Microsoft HoloLens 2; barcode reading; healthcare technology; medical vials; mixed reality; patient safety; vial detection.

Figure 1.

The trade-off between the distance and size of the code for optimal detection.

Figure 2.

Represents the sample vial images with barcodes of different sizes and vials with a similar look.

Figure 3.

The step-by-step process of the code scanning. Hololens 2 camera captures the barcode of the vial. If it is unable to do so, then the scanner attached will get the code and request its database to get related information.

Figure 4.

The image represents a user scanning a barcode on a small package using HoloLens 2, a user scanning a barcode on a whiteboard with HoloLens 2, and a barcode scanner with various vials used for testing.

Figure 5.

Flowchart of the vial detection process using the mixed reality technology.

Figure 6.

Represents the results of barcode size and detection distance. Blue color samples were detected by the MR device only. The red one was detected by an assisted Bluetooth scanner, which attached to the MR device.

Figure 7.

Represents the results after scanning the barcode. Scanning time varies for the Hololens camera based on the barcode size and camera resolution.

Figure 8.

Represents the visualization of frame rate in different lighting setup for Hololens 2 camera.

Figure 9.

Represents the visualization of detection accuracy for different camera resolution of Hololens 2.