GPU Implementation of a Gas-Phase Chemistry Solver in the CMAQ Chemical Transport Model

Abstract

The Community Multiscale Air Quality (CMAQ) model simulates atmospheric phenomena, including advection, diffusion, gas-phase chemistry, aerosol physics and chemistry, and cloud processes. Gas-phase chemistry is often a major computational bottleneck due to its representation as large systems of coupled nonlinear stiff differential equations. We leverage the parallel computational performance of graphics processing unit (GPU) hardware to accelerate the numerical integration of these systems in CMAQ's CHEM module. Our implementation, dubbed CMAQ-CUDA, in reference to its use in the Compute Unified Device Architecture (CUDA) general purpose GPU (GPGPU) computing solution, migrates CMAQ's Rosenbrock solver from Fortran to CUDA Fortran. CMAQ-CUDA accelerates the Rosenbrock solver such that simulations using the chemical mechanisms RACM2, CB6R5, and SAPRC07 require only 51%, 50%, or 35% as much time, respectively, as CMAQv5.4 to complete a chemistry time step. Our results demonstrate that CMAQ is amenable to GPU acceleration and highlight a novel Rosenbrock solver implementation for reducing the computational burden imposed by the CHEM module.

Figure 1

Schematic of CMAQ science process time stepping structure. SCIPROC is the Fortran routine that calls the science process routines. VDIFF is vertical diffusion. HADV is horizontal advection. ZADV is vertical advection. HDIFF is horizontal diffusion. CLDPROC denotes cloud processes. CHEM is gas-phase chemistry. AERO denotes aerosol processes. Times t_n and t_n+1 are the nth and subsequent synchronization time steps, respectively.

Figure 2

Pie chart of module timing burdens per time step for an 8-core simulation using the ROS3 solver and CB6R5 chemical mechanism with aero7 SOA treatment and standard cloud chemistry for the CMAQ 2018 12NE3 benchmark scenario.

Figure 3

Example schematic of domain subdivision and mapping to MPI processes. Each MPI processes is given its own CPU core and shares GPUs as instructed within our CMAQ-CUDA runtime configuration.

Figure 4

Schematic of CMAQ ROS3 code structure. Arrows represent data transfer. All routines are on the CPU.

Figure 5

Schematic of CMAQ-CUDA ROS3 code structure. Arrows represent data transfer.

Figure 6

Boxplots of CHEM module timing per synchronization time step for CMAQv5.4 and CMAQ-CUDA across three mechanisms.

Figure 7

Bar plots of total simulation walltime for CMAQv5.4 and CMAQ-CUDA across three mechanisms.

Figure 8

CMAQ science process module timing per synchronization time step as block size varies for CB6R5.

Figure 9

Maps of MB averaged in time for three key species in CB6R5. The left column is the MB between full simulations, the right column is the MB between simulations run with only the CHEM module active.

Figure 10

NME against median concentration for all species and all layers in the CB6R5 CHEM-only simulation. Each species appears 35 times, once for each vertical layer. NME is averaged in space (per layer) and time. Our 1% threshold is marked.

Figure 11

NME for the surface layer across the benchmark and three mechanisms. Each member of the distributions is a different species’ NME averaged in space and time. Highest errors are annotated with the corresponding species and its NME value. Our 1% threshold is marked.

Figure 12

NME for binned vertical layers across three mechanisms. Each member of the distributions is a different species’ NME averaged in space and time for the corresponding layer, such that each box plot has 5 members for each species: one for each layer in the corresponding bin. Our 1% threshold is marked.