Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Aug 5;40(21):1919-1930.
doi: 10.1002/jcc.25840. Epub 2019 Apr 17.

Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations

Jaewoon Jung  1 , Wataru Nishima  2   3 , Marcus Daniels  2 , Gavin Bascom  4 , Chigusa Kobayashi  1 , Adetokunbo Adedoyin  2 , Michael Wall  1 , Anna Lappala  2 , Dominic Phillips  2 , William Fischer  2 , Chang-Shung Tung  2 , Tamar Schlick  4 , Yuji Sugita  1 , Karissa Y Sanbonmatsu  2   3
Affiliations

Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations

Jaewoon Jung et al. J Comput Chem. .

Abstract

The growing interest in the complexity of biological interactions is continuously driving the need to increase system size in biophysical simulations, requiring not only powerful and advanced hardware but adaptable software that can accommodate a large number of atoms interacting through complex forcefields. To address this, we developed and implemented strategies in the GENESIS molecular dynamics package designed for large numbers of processors. Long-range electrostatic interactions were parallelized by minimizing the number of processes involved in communication. A novel algorithm was implemented for nonbonded interactions to increase single instruction multiple data (SIMD) performance, reducing memory usage for ultra large systems. Memory usage for neighbor searches in real-space nonbonded interactions was reduced by approximately 80%, leading to significant speedup. Using experimental data describing physical 3D chromatin interactions, we constructed the first atomistic model of an entire gene locus (GATA4). Taken together, these developments enabled the first billion-atom simulation of an intact biomolecular complex, achieving scaling to 65,000 processes (130,000 processor cores) with 1 ns/day performance. Published 2019. This article is a U.S. Government work and is in the public domain in the USA.

Keywords: 3D modeling; GENESIS MD software; biomolecular simulation; high performance computing.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
Structure of GATA4 gene, consisting of 83 kilobases of double stranded helical DNA wrapped around 427 nucleosomes. Protein tails used for programming gene expression protrude from each nucleosome.
Figure 2.
Figure 2.
Automated clash detection: flow diagram of the operation of the clash correction script. Clashes correctable by a phosphate group rotation are those involving P, O5’, C5’, C4’, C3’, O3’ type atoms on the DNA backbone. All other clashes involve at least one atom in a protein side group, and can be corrected by a side group rotation. The vicinity of a clash is defined to be all atoms within a 6Å radius of the atoms involved in the clash.
Figure 3.
Figure 3.
(a) Non-bonded interaction scheme used in GENESIS 1.0–1.3 and (b) new nonbonded interaction scheme for Intel Xeon Phi processors.
Figure 4.
Figure 4.
FFT time of (a) 10243 and (b) 20483 PME grids using GENESIS on Oakforest-PACS
Figure 5.
Figure 5.
Benchmark performance of 230M and 1B system on Oakforest-PACS
See this image and copyright information in PMC

References

    1. McCammon JA; Gelin BR; Karplus M Nature 1977, 267(5612), 585–590. - PubMed
    1. Narumi T; Ohno Y; Okimoto N; Koishi T; Suenaga A; Futatsugi N; Yanai R; Himeno R; Fujikawa S; Taiji M Proceedings of the 2006 ACM/IEEE conference on Supercomputing, 2006, pp 49.
    1. Shaw DE; Deneroff MM; Dror RO; Kuskin JS; Larson RH; Salmon JK; Young C; Batson B; Bowers KJ; Chao JC; Eastwood MP; J. G; Grossman JP; Ho CR; J. ID; Kolossvary I; Kelpeis JL; Layman T; McLeavey MA; Moraes MA; Mueller R; Edward C; Shan Y; Spengler J; Theobald M; Towles B; Wang SC 34th Annual International Symposium on Computer Architecture (ISCA ‘07), 2007, pp 91–97.
    1. Shaw DE; Grossman J; Bank JA; Batson B; Butts JA; Chao JC; Deneroff MM; Dror RO; Even A; Fenton CH Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2014, pp 41–53.
    1. Shaw DE; Maragakis P; Lindorff-Larsen K; Piana S; Dror RO; Eastwood MP; Bank JA; Jumper JM; Salmon JK; Shan YB; Wriggers W Science 2010, 330(6002), 341–346. - PubMed

Publication types

LinkOut - more resources