OpenRBC: A Fast Simulator of Red Blood Cells at Protein Resolution

Abstract

We present OpenRBC, a coarse-grained molecular dynamics code, which is capable of performing an unprecedented in silico experiment-simulating an entire mammal red blood cell lipid bilayer and cytoskeleton as modeled by multiple millions of mesoscopic particles-using a single shared memory commodity workstation. To achieve this, we invented an adaptive spatial-searching algorithm to accelerate the computation of short-range pairwise interactions in an extremely sparse three-dimensional space. The algorithm is based on a Voronoi partitioning of the point cloud of coarse-grained particles, and is continuously updated over the course of the simulation. The algorithm enables the construction of the key spatial searching data structure in our code, i.e., a lattice-free cell list, with a time and space cost linearly proportional to the number of particles in the system. The position and the shape of the cells also adapt automatically to the local density and curvature. The code implements OpenMP parallelization and scales to hundreds of hardware threads. It outperforms a legacy simulator by almost an order of magnitude in time-to-solution and >40 times in problem size, thus providing, to our knowledge, a new platform for probing the biomechanics of red blood cells.

Figure 1

(A) A canonical hexagonal triangular mesh of a biconcave surface representing the cytoskeleton network is used together with (B) the two-component CGMD RBC membrane model to reconstruct (C) a full-scale virtual RBC, which allows for a wide range of computational experiments. To see this figure in color, go online.

Figure 2

Shown here is the (A) flow chart and (B) typical wall time distribution of OpenRBC. To see this figure in color, go online.

Figure 3

(Left) Only cells in dark gray are populated by CG particles in a cell list on a rectilinear lattice. This results in a waste of storage and memory bandwidth. (Right) All cells are evenly populated by CG particles in a cell list based on the Voronoi diagram generated from centroids located on the RBC membrane. To see this figure in color, go online.

Figure 4

(A) Shown here is a Voronoi partitioning of a square as generated by centroids marked by the blue dots. (B) Shown here is a k-means (k = 3) clustering of a number of points on a two-dimensional plane. (C) Shown here is a vesicle of 32,673 CG particles partitioned into 2000 Voronoi cells. To see this figure in color, go online.

Figure 5

Thanks to the spatial locality ensured by reordering particles along the Morton curve (dashed line), we can simply divide the cells between two threads by their index into two patches each containing five consecutive Voronoi cells. The force between cells from the same patch is computed only once using Newton’s third law, whereas the force between cells from different patches is computed twice on each side. To see this figure in color, go online.

Figure 6

(A) Shown here is the vesiculation procedure of a miniature RBC. (B) The instantaneous fluctuation of a full-size RBC in OpenRBC compares to that from experiments (28, 29, 30). Microscopy image is reprinted with permission from Park et al. (28). To see this figure in color, go online.

Figure 7

Shown here is the scaling of OpenRBC across physical cores and NUMA domains when simulating an RBC of 3,200,000 particles. To see this figure in color, go online.