High-Throughput Line Buffer Microarchitecture for Arbitrary Sized Streaming Image Processing

Abstract

Parallel hardware designed for image processing promotes vision-guided intelligent applications. With the advantages of high-throughput and low-latency, streaming architecture on FPGA is especially attractive to real-time image processing. Notably, many real-world applications, such as region of interest (ROI) detection, demand the ability to process images continuously at different sizes and resolutions in hardware without interruptions. FPGA is especially suitable for implementation of such flexible streaming architecture, but most existing solutions require run-time reconfiguration, and hence cannot achieve seamless image size-switching. In this paper, we propose a dynamically-programmable buffer architecture (D-SWIM) based on the Stream-Windowing Interleaved Memory (SWIM) architecture to realize image processing on FPGA for image streams at arbitrary sizes defined at run time. D-SWIM redefines the way that on-chip memory is organized and controlled, and the hardware adapts to arbitrary image size with sub-100 ns delay that ensures minimum interruptions to the image processing at a high frame rate. Compared to the prior SWIM buffer for high-throughput scenarios, D-SWIM achieved dynamic programmability with only a slight overhead on logic resource usage, but saved up to 56 % of the BRAM resource. The D-SWIM buffer achieves a max operating frequency of 329.5 MHz and reduction in power consumption by 45.7 % comparing with the SWIM scheme. Real-world image processing applications, such as 2D-Convolution and the Harris Corner Detector, have also been used to evaluate D-SWIM's performance, where a pixel throughput of 4.5 Giga Pixel/s and 4.2 Giga Pixel/s were achieved respectively in each case. Compared to the implementation with prior streaming frameworks, the D-SWIM-based design not only realizes seamless image size-switching, but also improves hardware efficiency up to 30 × .

Keywords: D-SWIM; FPGA; high-throughput; line buffer; low-latency; streaming architecture.

Figure 1

A streaming architecture example for image processing with a 2D pattern. The architecture has three components: buffer, operator, and interconnections.

Figure 2

Motivation for arbitrary sized image processing: (a) user-defined Region of Interest (ROI) processing; (b) arbitrary sized image processing in cloud computing.

Figure 3

(a) Shows that the image lines are sequenced into a stream and then clipped to multi-pixel blocks; FPGA accepts one block in each cycle. (b) Shows the general pixel buffer in which the BRAM-misalignment issue occurs. (c) Shows the SWIM buffer avoids the BRAM-misalignment using specific BRAM partition.

Figure 4

Overview of D-SWIM framework.

Figure 5

(a) shows LB write behavior with conventional BRAM usage. (b) shows LB write behavior with the byte-wise write enable signal using BRAM primitive instantiation.

Figure 6

Example of buffer load and store with the line-rolling behavior.

Figure 7

(a) shows an example of the block storage pattern with parameters $N_{line}=44$, $N_{blk}=16$, and $H=3$. (b) shows the buffer instruction list for achieving the access pattern in (a).

Figure 8

D-SWIM workflow with dynamic programming for arbitrary sized image processing.

Figure 9

The D-SWIM buffer is composed of $\mathtt{LBs}$ and $\mathtt{Controller}$. Each BRAM in the LB is equipped with an $\mathtt{Address}\mathtt{Counter}$ to manage the write address. It performs address incrementation or reset according to the signal on the controller bus. The $\mathtt{Addr}\mathtt{MUX}$ allows the write addresses to be broadcasted during a block write operation of a specific LB as the read addresses of the other LBs for block loading.

Figure 10

Buffer-write logic.

Figure 11

Buffer-read logic.

Figure 12

(a) shows the $f_{max}$ of D-SWIM and SWIM designs with the configurations in Table 3. (b) shows the power consumption of D-SWIM and SWIM, with the breakdown of static and dynamic power.

Figure 13

D-SWIM-based architecture for Conv2D ($3\times 3$ window).

Figure 14

D-SWIM-based architecture for HC detector ($3\times 3$ window).