Hardware-Assisted Low-Latency NPU Virtualization Method for Multi-Sensor AI Systems

Abstract

Recently, AI systems such as autonomous driving and smart homes have become integral to daily life. Intelligent multi-sensors, once limited to single data types, now process complex text and image data, demanding faster and more accurate processing. While integrating NPUs and sensors has improved processing speed and accuracy, challenges like low resource utilization and long memory latency remain. This study proposes a method to reduce processing time and improve resource utilization by virtualizing NPUs to simultaneously handle multiple deep-learning models, leveraging a hardware scheduler and data prefetching techniques. Experiments with 30,000 SA resources showed that the hardware scheduler reduced memory cycles by over 10% across all models, with reductions of 30% for NCF and 70% for DLRM. The hardware scheduler effectively minimized memory latency and idle NPU resources in resource-constrained environments with frequent context switching. This approach is particularly valuable for real-time applications like autonomous driving, enabling smooth transitions between tasks such as object detection and route planning. It also enhances multitasking in smart homes by reducing latency when managing diverse data streams. The proposed system is well suited for resource-constrained environments that demand efficient multitasking and low-latency processing.

Keywords: hardware scheduler; multi-sensor; neural processing unit; prefetching; virtualization.

Figure 1

Examples of real-life applications of multi-sensor AI in various fields.

Figure 2

NPU virtualization operation flow. The symbol ‘#’ represents the number of cores.

Figure 3

TPU v4 architecture.

Figure 4

Hardware-assisted NPU virtualization system.

Figure 5

Comparison before and after the hardware scheduler when the burst size was changed.

Figure 6

Comparison before and after hardware scheduler application when the number of available SA changed.

Figure 7

Difference in memory cycles with and without a hardware scheduler.