Arrival & Registration

Opening

• Mathis Bode (Forschungszentrum Jülich GmbH)

Recent Developments in Nek5000/RS

• Paul Fischer

We describe recent developments in the high-order open-source simulation package Nek5000/RS, which is designed to solve turbulent thermal-fluids applications on platforms ranging from laptops to exascale computers. We begin with strong-scaling design considerations and discuss scaling on pre-exascale platforms such as ORNL's V100-based Summit and ANL's A100-based Polaris platforms and on the ORNL's exascale platform, Frontier, which has 72,000+ AMD MI250X GCDs. We discuss new features for Nek5000/RS, including the reduced-order modeling package, NekROM, developed by Kento Kaneko (MIT) and Ping-Hsuan Tsai (V.Tech) and MHD support in NekRS, which is being developed by ANL summer student, Yichen Guo (V.Tech). Several examples are presented for each.

nekRS Overview & Status Quo

• Stefan Kerkemeier

Exascale fission and fusion applications

• Elia Merzari (Penn State)

Advanced nuclear energy holds promise as a reliable, carbon-free energy source capable of meeting our nation's commitments to addressing climate change. A wave of investment in fission and fusion power within the United States and around the world indicates an important maturation of academic research projects into the commercial space. The design, certification, and licensing of novel reactor concepts pose formidable hurdles to the successful deployment of new technologies. The high cost of integral-effect nuclear experiments necessitates the use of high-fidelity numerical simulations to ensure the viability of nuclear energy in a clean energy portfolio. Building on our previous work, we will target simulations significantly larger than competing work in our field, and only with capability computing and exascale-level resources can these insights be gained. We NekRS, a GPU-oriented version of the Nek5000 code, to scale to the full Frontier machine. In particular, We discuss several high-fidelity simulation capabilities developing unprecedented insight into large-scale multi-physics phenomena. We discuss full-core hybrid Reynolds Averaged Navier Stokes (RANS) and Large Eddy Simulation (LES) of fission reactors conducted on Frontier. Simulation of unprecedented scale have also been conducted on a fusion energy systems (CHIMERA).

Boundary layers of thermal convection at very high Rayleigh numbers

• Roshan John Samuel (TU Ilmenau)

We perform simulations of Rayleigh-Bénard convection (RBC) at Rayleigh numbers ranging from 10⁵ to 10¹² and a fixed Prandtl number of 0.7. To simulate the canonical RBC setup with infinite horizontal extents, we employ a Cartesian box of aspect ratio 4 and periodic sides. We use the GPU accelerated spectral element solver, NekRS, on the GPU cluster, JUWELS Booster, at Jülich. The excellent scalability of NekRS is demonstrated by the fact that at the highest Ra of 10¹², with a grid of nearly 47 billion points, we report our statistics over 40 free-fall time units within steady-state. This simulation used the full-capacity of JUWELS Booster at nearly 3400 A100 GPUs. These high resolution simulations have enabled us to study the fine-structure of the boundary layers, identify shear-dominated and plume-dominated regions of the boundary layer flow, and evaluate their effects on heat and momentum transport. Another interesting outcome is that the area-fraction of these two regions is constant for the full range of Rayleigh numbers considered here. Finally, we compare the mean velocity profiles with Blasius profile to probe for signatures of flat-plate boundary layer flow.

Gridding complex geometries for spectral element method simulations

• Catherine Mavriplis (University of Ottawa)

Despite very successful complex geometry calculations using Nek over the years, gridding complex geometry cases remains a challenge. I will present some recent complex geometry cases using Nek and discuss possible new methodologies to simplify the gridding process. First, h-p adaptivity can be used to develop a suitable grid, starting from a fairly coarse mesh. However, the dynamic adaptive process quickly leads to imbalances in large scale computing. A load balancing algorithm for the hp-adaptive process will be presented along with scaling tests on both CPUs and GPUs. I will also discuss the treatment of curvilinear geometries using splines, mappings and immersed boundaries and compare results and efficiencies.

Direct numerical simulations of turbulence produced by wave attractors in stratified and/or rotating systems

• Ilias Sibgatullin (ENS de Lyon)

The propagation of internal waves in continuously stratified or rotating fluids differs radically from those of more traditional wave flows. It is worth mentioning that the dispersion relation connects the frequency only with the direction relative to gravity or rotation and does not determine the wavelength. Additionally, wave packets propagate perpendicular to the phase velocity. The billiard-like behavior of such wave packets in closed systems results in attracting trajectories. On these trajectories, the wave amplitude increases significantly, making them the origins of the onset of instabilities and turbulence. For such trajectories in the case of internal waves, boundaries inclined relative to the vertical are necessary. Previously, we investigated the onset of initial instabilities and the development of turbulence against the background of wave attractors when the vertical and horizontal scales of the flow are approximately equal. This model constitutes one important case of natural flows, the other case is the large aspect ratio domains, where the horizontal scale is much larger than the vertical, and still the buoyancy effects in momentum balance can't be neglected. For viscous fluids such an evolution of geometry results in significant changes in dynamics including the concentration of total kinetic energy, temporal and spatial spectra.

Airway flow modelling using Nek5000: Insights for gas transport during high-frequency ventilation

• Chinthaka Jacob (École Normale Supérieure de Lyon, CNRS, Laboratoire de physique; School of Engineering, Swinburne University of Technology)

High-frequency ventilation (HFV) is a medical ventilation technique that uses fast yet shallow inflations, resulting in small peak pressures, thereby protecting lungs from over-distension. While several mechanisms have been proposed for gas transport during HFV, this process is still not well understood, and it is likely the treatment as it stands is sub-optimal. Nonlinear mean streaming and turbulent diffusion are two mechanisms with the potential to be further exploited for gas transport. The work presented here aims to characterize and quantify these mechanisms in geometries, and at parameters, which are relevant to the application of HFV. These mechanisms have been investigated systematically in models with varying complexity – in a single generation and multi-generation bifurcating tubes. The geometries of the models are constructed to model a portion of the approximately self-similar human airway so that the flow in different portions can be modeled by simply changing model parameters. These findings are then extrapolated to quantify the role of these gas transport mechanisms in the entire airway. Finally, the overview of a flow-splitting algorithm is presented to highlight its use in combining numerical simulations with clinical measurements.

NekRS in turbulent combustion research of renewable fuels

• Driss Kaddar (Technical University of Darmstadt)

Hydrogen and Ammonia-based fuels will play a pivotal role for future carbon-free combustion systems. Direct numerical simulation (DNS) plays a pivotal role in establishing comprehensive understanding of the complex interactions of turbulence and the flame and forms the basis on which novel combustion models can be developed. We will be presenting the current activities on turbulent combustion DNS with nekRS/nekCRF at TU Darmstadt in collaboration with Jülich Supercomputing Centre and highlight contributions with potential interest to the community.

Spectral Element Dispersion for Coarse Meshes

• Nicholas Christensen

It is well known that Nek5000/RS's spectral element method (SEM) delivers spectacular convergence in the limit that the solution is well-resolved. Less understood, however, is the behavior of the SEM for marginally-resolved solutions that are frequently encountered in practice, particularly in the case of large-eddy simulations, where the turbulence is inherently under-resolved. We present an extensive study of 1D dispersion error for the SEM at varying wavenumbers and varying degrees of $h$- and $p$-refinement, with the principal parameter being the number of points per wavelength (PPW). The results illustrate some surprising behaviors, particularly at low PPW. We put these results into context with other studies and other discretizations, including high-order DG, and we suggest simple error mitigation strategies that can lead to improved performance for SEM advection in higher space dimensions.

Exascale @ JSC

Tour @ JSC

Social Event

Opening

• Mathis Bode (Forschungszentrum Jülich GmbH)

Exascale Advances with NekRS

• Misun Min (Argonne National Laboratory)

We discuss results of recent Exascale studies with NekRS. Through numerous simulation examples, we illustrate that NekRS sustains 80% parallel efficiency for local problem sizes, n/P, ranging from 3M points per MPI rank on OLCF's Frontier (2 ranks per AMD MI250X) to 5M points per rank on ALCF's NVIDIA A100-based Polaris. On 72,000 ranks of Frontier, NekRS sustains 0.39 TFLOPS per rank or a total of 28 PFLOPS for thermal hydraulics simulations in a full reactor core. In addition to nuclear energy applications, we describe recent developments in SEM-based wall modeled LES for atmospheric boundary layer simulations relevant to wind energy applications. We also present several technical developments that are important to exascale workflows. These include meshing and mesh partitioning for large meshes having in excess of 1B spectral elements; in situ visualization advances that avoid writing multi-TB output files; and GPU-based interpolation utilities that essential for particle tracking and for support of overset grids. Performance scaling results are presented for each of these developments. NekRS development is supported by the US Department of Energy's Advanced Scientific Computing Research program.

Developments in Hybrid RANS/LES and Wall Modeling approaches in Nek

• Ananias Tomboulides (Aristotle University of Thessaloniki)

The implementation and performance of non-zonal hybrid Reynolds Averaged Navier-Stokes (RANS)/Large Eddy Simulation (HRLES) approaches in Nek based on the k-tau RANS model will be presented. Results will be compared with direct numerical simulations (DNS) for benchmark cases such as the turbulent flow over the Periodic Hill geometry and the Plane Asymmetric Diffuser. Wall modeling approaches will also be presented for RANS and LES with representative examples.

Calculating Lyapunov Exponents with nek5000

• Janet Scheel (Occidental College)

The calculation of Lyapunov Exponents by using the perturbation solver in nek5000 will be discussed. This work was started by Anand Jayaraman and Paul Fischer (Jayaraman, et.al., PRE 2006) and has been used over the years to better understand chaos and turbulence in Rayleigh-Benard convection. Leading order Lyapunov exponents were calculated for small systems and moderate Rayleigh number to prove that they are chaotic (JS and Cross, PRE 2006). They were also helpful for determining onset states for intermediate-sized systems (Yu, et.al, Phys. of Fluids 2017), The leading order Lyapunov eigenvector can also provide insight into the nature of the chaos. For example an analysis of the leading order Lyapunov eigenvector for constant heat-flux driven convection was used to support the supergranule aggregation phenomena observed in these systems (Vieweg, et.al, Phys Rev Research 2021).

NekRS for fluid simulations in fusion multiphysics

• Rupert Eardley-Brunt (United Kingdom Atomic Energy Authority)

In order to accelerate development of magnetic confinement fusion from experimental tokamaks to power plants, detailed computational multiphysics approaches are being developed to enable predictive modelling and in silico design. A key step towards this goal is identifying a highly scalable computational fluid dynamics code to tackle the large, challenging fluids problems involved. NekRS is being explored for this purpose, building towards application to cases such as coolant flows in complex pipe systems and designs for components including the hypervapotron and tritium breeder pins, with the ultimate aim of connecting these systems to multiphysics simulations in MOOSE using Cardinal. In addition, a challenging problem in fusion is modelling the flows of liquid metals, which feature in some tritium breeder and divertor designs, and their strong magnetohydrodynamic coupling to the magnetic fields used to confine the plasma. Numerical modelling of liquid-metal MHD is generally less developed than conventional CFD, and NekRS is being considered as a potential route to highly scalable liquid-metal MHD simulation. This talk summarises progress with learning to use NekRS, as well as Cardinal for coupling into MOOSE, and outlines future plans for application to fusion-relevant problems, including coolant flows, liquid metal breeder blanket analysis, and multiphysics interactions.

Numerical Study of Flow Past a Wall-Mounted Dolphin Dorsal Fin at Low Reynolds Number

• Zhonglu Lin (Xiamen University)

Hydrodynamics of dolphin swimming has long been an attractive topic, yet few studies have focused on the function of its iconic dorsal fin. Here, we present high fidelity numerical simulations for flow around a 3-D wall-mounted dolphin dorsal fin based on a scanning from a real dolphin. The spectral element method is applied through NEK5000 to ensure high accuracy and efficiency of the simulations, as well as the application of the unstructured hex mesh. Six cases are studied at attack angle $AoA = 0, 60^{\circ} $ and Reynolds number $Re = 691, 1000, 2000$ with the analysis of the force coefficient and the 3-D flow characteristics.

SYNERGISTIC EFFECTS OF TURBULENCE AND THERMODIFFUSIVE INSTABILITIES ON EARLY FLAME KERNEL PROPAGATION IN A LEAN HYDROGEN-AIR MIXTURE

• Ioannis Kavroulakis (Aristotle University of Thessaloniki)

A comprehensive series of direct numerical simulations (DNS) is performed to investigate the early flame kernel development (EFKD) in a lean premixed H2-air mixture in decaying homogeneous isotropic turbulence and engine-relevant thermodynamic conditions. Systematic variations of turbulent intensity and integral length scale were assessed, resulting in Karlovitz number between 1.9 and 21. The main objective of this study is to explore the variations induced by turbulence during the EFKD phase in lean hydrogen-air mixtures across various turbulent regimes and assess their influence on the evolution of flame kernels. The study unveils a significant influence of the global stretch factor during the initial phases of flame kernel evolution. However, as the post-ignition effects diminish, the dominant factor shifts towards the wrinkling of the flame front. Elevated values of turbulence intensity lead to increased flame convolution and small-scale wrinkling, while higher integral length values contribute to a smoother flame surface. Higher Karlovitz numbers correlate with intensified fuel consumption, driven by accelerated flame surface expansion from enhanced wrinkling and increased local consumption speed due to differential diffusion effects.

NUMERICAL INVESTIGATION OF SOOT FORMATION IN A LABORATORY-SCALE RICH-QUENCH-LEAN SWIRL BURNER USING NEK5000

• Dimitris Papageorgiou (Aristotle University of Thessaloniki)

The primary objective of this study is to employ a DNS framework to replicate the intricate phenomena of a laboratory-scale soot configuration, specifically the UCAM RQL (Rich-burn Quick-quench Lean-burn) Burner . The moment-based soot model MOMIC was integrated into the high-order CFD code Nek5000 with the development of an in-house plugin, establishing a framework to assess the accuracy and predictive capabilities of the code in adequately evaluating parameters strongly related with the formation of soot particles and delve into the underlying turbulence-chemistry-soot interactions. The inlet of the laboratory scale aero-engine UCAM RQL burner is composed of two concentric pipes, where ethylene flows through the inner pipe and the primary air flows through the outer pipe in a swirling motion imposed by an axial swirler, resulting in intense turbulent mixing inside the burning chamber. The reactive Navier-Stokes equations in the formulation of low-Mach number regime and Nek5000’s reactive plug-in was employed to calculate the thermal and transport properties along the chemical source terms. A 62-species reduced version of the ABF mechanism was utilized as the chemical mechanism in order to incorporate large soot PAH precursors up to A4.

Dispersed Microbubble-Laden Turbulent Flow Based on High-Order Euler-Lagrange Approach

• Byeong-Cheon Kim (University of Ulsan)

To resolve multiphase flow, specifically dispersed phase flow, tracking the dispersed phase's trajectory is crucial. The Euler-Lagrange approach is adopted to predict the interaction between the dispersed phase (microbubble) and the continuous phase (turbulence). The Lagrangian tracking code, ppiclF (parallel particle-in-cell library written in Fortran), and the spectral element method code, Nek5000, are combined to simulate microbubble-laden turbulent flows. In this presentation, the developed microbubble models and microbubble dynamics in the turbulent channel flow will be introduced. Turbulent quantities such as turbulent boundary layer and Reynolds stresses are compared with respect to the size and number of bubbles. Additionally the drag reduction mechanism by microbubbles is analyzed.

Flow Characteristics in Two-Dimensional and Three-Dimensional Thermo-Diffusive Unstable Flames

• Hamid Kavari (Sapienza University of Rome)

Hydrogen-based fuel burning faces challenges due to intrinsic premixed flame instabilities that affect flame morphology. This study introduces a three-dimensional (3D) Direct Numerical Simulation (DNS) dataset using a low-Mach formulation and a deficient reactant thermochemical model with NekRS, comparing flame characteristics to a well-established two-dimensional (2D) dataset [1]. Comparing the 2D and 3D flames reveals similar finger-like structures, but 3D flames exhibit 10% higher superadiabatic temperatures at cusps due to higher local curvatures enhancing local differential diffusion effects. The 3D flames also demonstrate higher mean consumption speeds, with the smallest 3D flame being faster than all 2D cases and 2.5 times its 2D counterpart's speed. The 3D flames exhibit a narrower range of flame curvature but higher displacement speeds and tangential strain rates. Overall, 3D flames show more positive stretch, leading to higher consumption speeds and reaction rates. The study concludes that 3D flames exhibit higher temperature peaks and faster reactions compared to 2D flames, attributed to increased thermal diffusion and more positive flame stretch characteristics in 3D. Reference [1] Creta, Francesco, et al. Combustion and Flame 216 (2020): 256-270.

Topology Optimization of Roughness Elements in Boundary Layers

• Harrison Nobis (KTH)

This article applies density-based topology optimization in order to design roughness elements capable of generating stable streaks to damp the growth of Tollmien-Schlichting (TS) waves in a boundary layer. First a steady baseflow is established, then the unsteady linearized Navier-Stokes equations are evolved to assess the spatial growth of the TS waves across the flat plate. The optimization procedure aims to minimize the TS wave amplitude at a given downstream position while a novel constraint is used promoting a stable baseflow. This method has been applied to three initial material distributions yielding three distinct and novel designs capable of damping the downstream growth of the TS wave significantly more than a reference Minature Vortex Generator (MVG) of comparable size. The optimized designs and streaky baseflows they induce are then studied a posteriori using an energy budget analysis and local stability analysis.

Ongoing NekRS Documentation Efforts

• Jez Swann

Closing

• Mathis Bode (Forschungszentrum Jülich GmbH)

Social Event

Opening

• Mathis Bode (Forschungszentrum Jülich GmbH)

Hands-on

• Jez Swann

Hands-on

• Jez Swann

Closing

• Mathis Bode (Forschungszentrum Jülich GmbH)