In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst

Abstract

In situ visualization on high-performance computing systems allows us to analyze simulation results that would otherwise be impossible, given the size of the simulation data sets and offline post-processing execution time. We develop an in situ adaptor for Paraview Catalyst and Nek5000, a massively parallel Fortran and C code for computational fluid dynamics. We perform a strong scalability test up to 2048 cores on KTH's Beskow Cray XC40 supercomputer and assess in situ visualization's impact on the Nek5000 performance. In our study case, a high-fidelity simulation of turbulent flow, we observe that in situ operations significantly limit the strong scalability of the code, reducing the relative parallel efficiency to only $\approx 21\%$ on 2048 cores (the relative efficiency of Nek5000 without in situ operations is $\approx 99\%$ ). Through profiling with Arm MAP, we identified a bottleneck in the image composition step (that uses the Radix-kr algorithm) where a majority of the time is spent on MPI communication. We also identified an imbalance of in situ processing time between rank 0 and all other ranks. In our case, better scaling and load-balancing in the parallel image composition would considerably improve the performance of Nek5000 with in situ capabilities. In general, the result of this study highlights the technical challenges posed by the integration of high-performance simulation codes and data-analysis libraries and their practical use in complex cases, even when efficient algorithms already exist for a certain application scenario.

Keywords: Computational fluid dynamics; High-performance computing; In situ visualization.

Fig. 1

Workflow of in situ visualization of an IsoLambda2 simulation using a Catalyst adaptor and a pipeline script. The visualization pipeline describes the configuration and steps (such as how data is processed and rendered) using the ParaView Python interface

Fig. 2

Structure of the in situ adaptor implemented in this project. A more detailed description is provided in the repository documentation at https://github.com/KTH-Nek5000/InSituPackage

Fig. 3

Detail of the mesh in the proximity of the NACA4412 airfoil and (insert) side view of the computational domain. Note that in the side view only the spectral elements are shown

Fig. 4

Iso-surface of the ${\mathit{\lambda }}_{\mathbf{2}}$ criterion [12] to identify near-wall vortical structures for the 3D turbulent flow around a NACA 4412 wing section

Fig. 5

Average execution time per process on each time step over the entire simulation for $\mathbf{n}=\mathbf{1000}$ in log scale. Different numbers of scaling configurations, 256 (green), 512 (purple), 1024 (orange), and 2048 (brown) cores, are used in each test case (color figure online)

Fig. 6

Scaling of the mean wall-time per process when scaling the number of processes. The time steps with and without in situ operations are marked in blue and red respectively. A log scale is used and error bars indicate the 95% confidence interval

Fig. 7

Execution time per time step for the in situ visualization pipeline, for MPI rank 0 (pink) and MPI rank 1 (yellow). Results from other ranks are not reported as they behave similarly to rank 1. A log scale is used and error bars indicate the 95% confidence interval.

Fig. 8

a Profiling of a full Nek5000 simulation using Arm MAP allows us to distinguish between compute and MPI workloads. b The time distribution reveals that MPI accounts for approximately 70.7% of the execution time when co-processing is active (where 41.4% is from co-processing). c We investigate the source of the bottleneck by expanding the call stack of the in situ processing function and find that d a MPI_Allreduce is taking 7.8% of the time. e However, a MPI_Waitany that is used in the image composition inside the ICE-T library (icetRadixkrCompose) accounts for 33.0% of the time