DOE Gordon Bell Prize

This novel work presents a scalable, multimodal workflow for protein design that trains an LLM to generate protein sequences, computationally evaluates the generated sequences, and then exploits them to fine-tune the model.  Direct Preference Optimization steers the LLM toward the generation of preferred sequences, and enhanced workflow technology enables its efficient execution. A 3.5B and a 7B model demonstrate scalability and exceptional mixed precision performance of the full workflow on ALPS, Aurora, Frontier, Leonardo and PDX.

Affiliations: Argonne National Laboratory, California Institute of Technology, CINECA, NVIDIA Corp., University of California at Berkeley

This team developed an output accuracy-preserving method for exploiting low-precision data types in matrix computations and used it to create a highly efficient Cholesky-based solver. Their tile-centric adaptive precision matrix operations, and task-based execution, enabled the largest-ever Genome-Wide Association Studies (GWAS) of 305K patients from a real data set, using a multivariate approach to identify genetic risk factors.  Its outstanding scaling and very high mixed-precision performance was demonstrated on 8,100 GPUs of Alps, 36,100 GPUs of Frontier, 4096 GPUs of Leonardo, and 18432 GPUs of Summit.

Affiliations: KAUST, Massachusetts Institute of Technology, Saint Louis University, NVIDIA Corp.

This team has created an Embedded Atom Method (EAM)-based molecular dynamics code that exploits the ultra-fast communication and high memory bandwidth afforded by the 850,000 core-Cerebras Wafer-Scale Engine. It attains perfect weak scaling across the full system for grain boundary problems involving copper, tungsten and tantalum atoms, and can extend to multiple wafers. For problems up to 800,000 atoms, it calculates significantly more timesteps per second than EAM in LAMMPS on Quartz and Frontier, directly benefiting the modeling of phenomena that emerge at long timescales.

Affiliations: Cerebras Systems, Sandia National Laboratories, Lawrence Livermore National Laboratory, Los Alamos National Laboratory

This work presents AxoNN, a scalable, portable, open-source framework for training and fine-tuning large language models (LLMs) and describes optimizations applied by the team that enable it to handle LLMs with hundreds of billions to trillions of parameters. They provide results of a study on the potential for the memorization of training data by large LLMs, using AxoNN on up to 405 billion parameters on Frontier.  Their evaluations show the exceptional scaling and performance attained when training GPT-style transformer models with up 640 billion parameters on Alps, Frontier and Perlmutter.

Affiliations: University of Maryland, Max Planck Institute for Information Systems, University of California, Berkeley

This work describes a novel approach for accurately simulating complex biochemical phenomena via biomolecular-scale Ab Initio Molecular Dynamics simulations at quantum molecular wave function level.  Multiple algorithmic innovations were employed to overcome the computational challenges posed by the use of the second-order Moller-Plesset perturbation theory. Evaluated using biomolecules with up to 2,043,328 electrons, the code exhibits high parallel efficiencies on the full Perlmutter and Frontier systems and sustained an unprecedented 59% of FP64 peak performance on Frontier.

Affiliations: University of Melbourne, Australian National University, AMD Inc., Oak Ridge National Laboratory

To advance our understanding of global climate change at the local scale there are both massive computational and data storage challenges. This work leverages the state-of-the-art in high-performance computing to develop a scalable implementation of an exascale climate emulator that illustrates the potential of supercomputers to address the challenges of climate modeling at ultra-high resolution. They also demonstrate the potential for significant savings in the petabytes of storage required for storing and sharing climate simulations. Their approach uses the Spherical Harmonic Transform (SHT) for modeling spatio-temporal climate data that can accommodate climate data sourced from various spatial resolutions. The exascale climate emulator developed in this study holds significant potential for the climate community, advancing climate research and policy making by making high resolution climate data more readily available so that we can better address the threat of global climate change.

Affiliations: Extreme Computing & Statistics & Earth Science, King Abdullah University of Science and Technology, KSA Computational and Information Sciences Lab, NSF National Center for Atmospheric Research, USA NVIDIA, USA Department of Computer Science, Saint Louis University, USA Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, USA

The combination of rapidly advancing AI techniques and abundant simulation data from multi-model ensemble projects such as CMIP6 forms the basis for ORBIT the Oak Ridge Base Foundation Model for Earth System Predictability.

They designed ORBIT to scale up 113 billion parameters, the largest dense vision transformer to date, and were able to incorporate a record setting 91 channels of climate variables. ORBIT was pre-trained on 10 different CMIP6 datasets that included 1.2 million observation data points. ORBIT was able to achieve impressive performance during training delivering up to 1.6 exaFLOPS/684 PFLOPS for the 10/113 billion parameter models on 49,152 Frontier AMD GPUs using mixed single BFLOAT16 precision using a novel Hybrid Sharded Tensor-Data Orthogonal Parallelism (Hybrid-STOP).

Importantly, this approach is not reliant on specialized HPC architectures, making it applicable on a broad range of HPC systems. Thus, ORBIT represents a significant step forward in advancing our Earth system modeling capabilities.

Affiliations: Oak Ridge National Laboratory, Oak Ridge, United States, AMD Research and Advanced Development, Santa Clara, United States