Nek User MeetingIn-Person Event

Jul 29, 2024, 8:00 AM → Jul 31, 2024, 5:00 PM Europe/Berlin

Jülich Supercomputing Centre

Mathis Bode (Forschungszentrum Jülich GmbH)

[On the left-hand side you have the opportunity to register for the event and submit a proposal for a short presentation.]

We are excited to announce the upcoming Nek User Meeting, a semi-regular event for the Nek Community, set to convene in Jülich for the first time. This gathering presents a unique opportunity to engage with Nek developers and users to discuss topics in High-Performance Computing (HPC) and Computational Fluid Dynamics (CFD), to discover the latest Nek advancements, and to have discussions in an informal and enjoyable setting.

Tracing its origins back to the mid-1980s, the Nek project continues to push the envelope in supercomputing. Its flagship codes, nek5000 and nekRS, have been instrumental in shaping the landscape of high-performance CFD.  Nek’s involvement in prestigious HPC initiatives, such as the Exascale Computing Project (ECP) and JUREAP, has set benchmarks in performance and served as a blueprint for state-of-the-art CFD computations. Notably, Nek codes are at the heart of several major computational projects at the Jülich Supercomputing Centre (JSC), as well as in the US and Asia.

The Nek User Meeting is poised to offer a dynamic platform for knowledge sharing, featuring a series of invited talks and contributed presentations that feature the latest findings within the user/developer community. Newcomers will benefit from an introductory session designed to acquaint them with the Nek ecosystem.

We look forward to seeing you in Jülich!

Mon, July 29

8:30 AM → 9:00 AM Arrival & Registration

9:00 AM → 9:10 AM Opening

Speaker: Mathis Bode (Forschungszentrum Jülich GmbH)

9:10 AM → 9:50 AM Recent Developments in Nek5000/RS

We describe recent developments in the high-order open-source simulation
package Nek5000/RS, which is designed to solve turbulent thermal-fluids
applications on platforms ranging from laptops to exascale computers. We begin
with strong-scaling design considerations and discuss scaling on pre-exascale
platforms such as ORNL's V100-based Summit and ANL's A100-based Polaris
platforms and on the ORNL's exascale platform, Frontier, which has 72,000+ AMD
MI250X GCDs. We discuss new features for Nek5000/RS, including the
reduced-order modeling package, NekROM, developed by Kento Kaneko (MIT) and
Ping-Hsuan Tsai (V.Tech) and MHD support in NekRS, which is being developed by
ANL summer student, Yichen Guo (V.Tech). Several examples are presented for
each.

Speaker: Paul Fischer

9:50 AM → 10:30 AM nekRS Overview & Status Quo

Speaker: Stefan Kerkemeier

10:30 AM → 11:00 AM Coffee Break

11:00 AM → 11:40 AM Exascale fission and fusion applications

Advanced nuclear energy holds promise as a reliable, carbon-free energy source capable of meeting our nation's commitments to addressing climate change. A wave of investment in fission and fusion power within the United States and around the world indicates an important maturation of academic research projects into the commercial space. The design, certification, and licensing of novel reactor concepts pose formidable hurdles to the successful deployment of new technologies. The high cost of integral-effect nuclear experiments necessitates the use of high-fidelity numerical simulations to ensure the viability of nuclear energy in a clean energy portfolio.

Building on our previous work, we will target simulations significantly larger than competing work in our field, and only with capability computing and exascale-level resources can these insights be gained. We NekRS, a GPU-oriented version of the Nek5000 code, to scale to the full Frontier machine.
In particular, We discuss several high-fidelity simulation capabilities developing unprecedented insight into large-scale multi-physics phenomena. We discuss full-core hybrid Reynolds Averaged Navier Stokes (RANS) and Large Eddy Simulation (LES) of fission reactors conducted on Frontier. Simulation of unprecedented scale have also been conducted on a fusion energy systems (CHIMERA).

Speaker: Elia Merzari (Penn State)

11:40 AM → 12:20 PM Boundary layers of thermal convection at very high Rayleigh numbers

We perform simulations of Rayleigh-Bénard convection (RBC) at Rayleigh numbers ranging from 10⁵ to 10¹² and a fixed Prandtl number of 0.7. To simulate the canonical RBC setup with infinite horizontal extents, we employ a Cartesian box of aspect ratio 4 and periodic sides. We use the GPU accelerated spectral element solver, NekRS, on the GPU cluster, JUWELS Booster, at Jülich. The excellent scalability of NekRS is demonstrated by the fact that at the highest Ra of 10¹², with a grid of nearly 47 billion points, we report our statistics over 40 free-fall time units within steady-state. This simulation used the full-capacity of JUWELS Booster at nearly 3400 A100 GPUs.

These high resolution simulations have enabled us to study the fine-structure of the boundary layers, identify shear-dominated and plume-dominated regions of the boundary layer flow, and evaluate their effects on heat and momentum transport. Another interesting outcome is that the area-fraction of these two regions is constant for the full range of Rayleigh numbers considered here. Finally, we compare the mean velocity profiles with Blasius profile to probe for signatures of flat-plate boundary layer flow.

Speaker: Roshan John Samuel (TU Ilmenau)

12:20 PM → 1:20 PM Lunch Break

1:20 PM → 1:40 PM Gridding complex geometries for spectral element method simulations

Despite very successful complex geometry calculations using Nek over the years, gridding complex geometry cases remains a challenge. I will present some recent complex geometry cases using Nek and discuss possible new methodologies to simplify the gridding process. First, h-p adaptivity can be used to develop a suitable grid, starting from a fairly coarse mesh. However, the dynamic adaptive process quickly leads to imbalances in large scale computing. A load balancing algorithm for the hp-adaptive process will be presented along with scaling tests on both CPUs and GPUs. I will also discuss the treatment of curvilinear geometries using splines, mappings and immersed boundaries and compare results and efficiencies.

Speaker: Dr Catherine Mavriplis (University of Ottawa)

1:40 PM → 2:00 PM Direct numerical simulations of turbulence produced by wave attractors in stratified and/or rotating systems

The propagation of internal waves in continuously stratified or rotating fluids differs radically from those of more traditional wave flows. It is worth mentioning that the dispersion relation connects the frequency only with the direction relative to gravity or rotation and does not determine the wavelength. Additionally, wave packets propagate perpendicular to the phase velocity. The billiard-like behavior of such wave packets in closed systems results in attracting trajectories. On these trajectories, the wave amplitude increases significantly, making them the origins of the onset of instabilities and turbulence. For such trajectories in the case of internal waves, boundaries inclined relative to the vertical are necessary.
Previously, we investigated the onset of initial instabilities and the development of turbulence against the background of wave attractors when the vertical and horizontal scales of the flow are approximately equal.
This model constitutes one important case of natural flows, the other case is the large aspect ratio domains, where the horizontal scale is much larger than the vertical, and still the buoyancy effects in momentum balance can't be neglected.
For viscous fluids such an evolution of geometry results in significant changes in dynamics including the concentration of total kinetic energy, temporal and spatial spectra.

Speaker: Ilias Sibgatullin (ENS de Lyon)

2:00 PM → 2:20 PM Airway flow modelling using Nek5000: Insights for gas transport during high-frequency ventilation

High-frequency ventilation (HFV) is a medical ventilation technique that uses fast yet shallow inflations, resulting in small peak pressures, thereby protecting lungs from over-distension. While several mechanisms have been proposed for gas transport during HFV, this process is still not well understood, and it is likely the treatment as it stands is sub-optimal. Nonlinear mean streaming and turbulent diffusion are two mechanisms with the potential to be further exploited for gas transport. The work presented here aims to characterize and quantify these mechanisms in geometries, and at parameters, which are relevant to the application of HFV.
These mechanisms have been investigated systematically in models with varying complexity – in a single generation and multi-generation bifurcating tubes. The geometries of the models are constructed to model a portion of the approximately self-similar human airway so that the flow in different portions can be modeled by simply changing model parameters. These findings are then extrapolated to quantify the role of these gas transport mechanisms in the entire airway.

Finally, the overview of a flow-splitting algorithm is presented to highlight its use in combining numerical simulations with clinical measurements.

Speaker: Chinthaka Jacob (École Normale Supérieure de Lyon, CNRS, Laboratoire de physique; School of Engineering, Swinburne University of Technology)

2:20 PM → 2:40 PM NekRS in turbulent combustion research of renewable fuels

Hydrogen and Ammonia-based fuels will play a pivotal role for future carbon-free combustion systems. Direct numerical simulation (DNS) plays a pivotal role in establishing comprehensive understanding of the complex interactions of turbulence and the flame and forms the basis on which novel combustion models can be developed. We will be presenting the current activities on turbulent combustion DNS with nekRS/nekCRF at TU Darmstadt in collaboration with Jülich Supercomputing Centre and highlight contributions with potential interest to the community.

Speaker: Driss Kaddar (Technical University of Darmstadt)

2:40 PM → 3:00 PM Spectral Element Dispersion for Coarse Meshes

It is well known that Nek5000/RS's spectral element method (SEM) delivers
spectacular convergence in the limit that the solution is well-resolved. Less
understood, however, is the behavior of the SEM for marginally-resolved
solutions that are frequently encountered in practice, particularly in the case
of large-eddy simulations, where the turbulence is inherently under-resolved.
We present an extensive study of 1D dispersion error for the SEM at varying
wavenumbers and varying degrees of $h$- and $p$-refinement, with the principal
parameter being the number of points per wavelength (PPW). The results
illustrate some surprising behaviors, particularly at low PPW. We put these
results into context with other studies and other discretizations, including
high-order DG, and we suggest simple error mitigation strategies that can lead
to improved performance for SEM advection in higher space dimensions.

Speaker: Nicholas Christensen

3:00 PM → 3:30 PM Coffee Break

3:30 PM → 4:30 PM Exascale @ JSC

4:30 PM → 5:30 PM Tour @ JSC

5:30 PM → 7:00 PM Social Event

Tue, July 30

9:00 AM → 9:10 AM Opening

Speaker: Mathis Bode (Forschungszentrum Jülich GmbH)

9:10 AM → 9:50 AM Exascale Advances with NekRS

We discuss results of recent Exascale studies with NekRS. Through numerous simulation examples, we illustrate that NekRS sustains 80% parallel efficiency for local problem sizes, n/P, ranging from 3M points per MPI rank on OLCF's Frontier (2 ranks per AMD MI250X) to 5M points per rank on ALCF's NVIDIA A100-based Polaris. On 72,000 ranks of Frontier, NekRS sustains 0.39 TFLOPS per rank or a total of 28 PFLOPS for thermal hydraulics simulations in a full reactor core. In addition to nuclear energy applications, we describe recent developments in SEM-based wall modeled LES for atmospheric boundary layer simulations relevant to wind energy applications. We also present several technical developments that are important to exascale workflows. These include meshing and mesh partitioning for large meshes having in excess of 1B spectral elements; in situ visualization advances that avoid writing multi-TB output files; and GPU-based interpolation utilities that essential for particle tracking and for support of overset grids. Performance scaling results are presented for each of these developments. NekRS development is supported by the US Department of Energy's Advanced Scientific Computing Research program.

Speaker: Misun Min (Argonne National Laboratory)

9:50 AM → 10:30 AM Developments in Hybrid RANS/LES and Wall Modeling approaches in Nek

The implementation and performance of non-zonal hybrid Reynolds Averaged Navier-Stokes (RANS)/Large Eddy Simulation (HRLES) approaches in Nek based on the k-tau RANS model will be presented. Results will be compared with direct numerical simulations (DNS) for benchmark cases such as the turbulent flow over the Periodic Hill geometry and the Plane Asymmetric Diffuser. Wall modeling approaches will also be presented for RANS and LES with representative examples.

Speaker: Ananias Tomboulides (Aristotle University of Thessaloniki)

10:30 AM → 11:00 AM Coffee Break

11:00 AM → 11:40 AM Calculating Lyapunov Exponents with nek5000

The calculation of Lyapunov Exponents by using the perturbation solver in nek5000 will be discussed. This work was started by Anand Jayaraman and Paul Fischer (Jayaraman, et.al., PRE 2006) and has been used over the years to better understand chaos and turbulence in Rayleigh-Benard convection. Leading order Lyapunov exponents were calculated for small systems and moderate Rayleigh number to prove that they are chaotic (JS and Cross, PRE 2006). They were also helpful for determining onset states for intermediate-sized systems (Yu, et.al, Phys. of Fluids 2017), The leading order Lyapunov eigenvector can also provide insight into the nature of the chaos. For example an analysis of the leading order Lyapunov eigenvector for constant heat-flux driven convection was used to support the supergranule aggregation phenomena observed in these systems (Vieweg, et.al, Phys Rev Research 2021).

Speaker: Janet Scheel (Occidental College)

11:40 AM → 12:20 PM NekRS for fluid simulations in fusion multiphysics

In order to accelerate development of magnetic confinement fusion from experimental tokamaks to power plants, detailed computational multiphysics approaches are being developed to enable predictive modelling and in silico design. A key step towards this goal is identifying a highly scalable computational fluid dynamics code to tackle the large, challenging fluids problems involved. NekRS is being explored for this purpose, building towards application to cases such as coolant flows in complex pipe systems and designs for components including the hypervapotron and tritium breeder pins, with the ultimate aim of connecting these systems to multiphysics simulations in MOOSE using Cardinal. In addition, a challenging problem in fusion is modelling the flows of liquid metals, which feature in some tritium breeder and divertor designs, and their strong magnetohydrodynamic coupling to the magnetic fields used to confine the plasma. Numerical modelling of liquid-metal MHD is generally less developed than conventional CFD, and NekRS is being considered as a potential route to highly scalable liquid-metal MHD simulation. This talk summarises progress with learning to use NekRS, as well as Cardinal for coupling into MOOSE, and outlines future plans for application to fusion-relevant problems, including coolant flows, liquid metal breeder blanket analysis, and multiphysics interactions.

Speaker: Mr Rupert Eardley-Brunt (United Kingdom Atomic Energy Authority)

12:20 PM → 1:20 PM Lunch Break

1:20 PM → 1:40 PM Numerical Study of Flow Past a Wall-Mounted Dolphin Dorsal Fin at Low Reynolds Number

Hydrodynamics of dolphin swimming has long been an attractive topic, yet few studies have focused on the function of its iconic dorsal fin. Here, we present high fidelity numerical simulations for flow around a 3-D wall-mounted dolphin dorsal fin based on a scanning from a real dolphin. The spectral element method is applied through NEK5000 to ensure high accuracy and efficiency of the simulations, as well as the application of the unstructured hex mesh. Six cases are studied at attack angle $AoA = 0, 60^{\circ} $ and Reynolds number $Re = 691, 1000, 2000$ with the analysis of the force coefficient and the 3-D flow characteristics.

Speaker: Zhonglu Lin (Xiamen University)

1:40 PM → 2:00 PM SYNERGISTIC EFFECTS OF TURBULENCE AND THERMODIFFUSIVE INSTABILITIES ON EARLY FLAME KERNEL PROPAGATION IN A LEAN HYDROGEN-AIR MIXTURE

A comprehensive series of direct numerical simulations (DNS) is performed to investigate the early flame kernel development (EFKD) in a lean premixed H2-air mixture in decaying homogeneous isotropic turbulence and engine-relevant thermodynamic conditions. Systematic variations of turbulent intensity and integral length scale were assessed, resulting in Karlovitz number between 1.9 and 21. The main objective of this study is to explore the variations induced by turbulence during the EFKD phase in lean hydrogen-air mixtures across various turbulent regimes and assess their influence on the evolution of flame kernels. The study unveils a significant influence of the global stretch factor during the initial phases of flame kernel evolution. However, as the post-ignition effects diminish, the dominant factor shifts towards the wrinkling of the flame front. Elevated values of turbulence intensity lead to increased flame convolution and small-scale wrinkling, while higher integral length values contribute to a smoother flame surface. Higher Karlovitz numbers correlate with intensified fuel consumption, driven by accelerated flame surface expansion from enhanced wrinkling and increased local consumption speed due to differential diffusion effects.

Speaker: Ioannis Kavroulakis (Aristotle University of Thessaloniki)

2:00 PM → 2:20 PM NUMERICAL INVESTIGATION OF SOOT FORMATION IN A LABORATORY-SCALE RICH-QUENCH-LEAN SWIRL BURNER USING NEK5000

The primary objective of this study is to employ a DNS framework to replicate the intricate phenomena of a laboratory-scale soot configuration, specifically the UCAM RQL (Rich-burn Quick-quench Lean-burn) Burner . The moment-based soot model MOMIC was integrated into the high-order CFD code Nek5000 with the development of an in-house plugin, establishing a framework to assess the accuracy and predictive capabilities of the code in adequately evaluating parameters strongly related with the formation of soot particles and delve into the underlying turbulence-chemistry-soot interactions. The inlet of the laboratory scale aero-engine UCAM RQL burner is composed of two concentric pipes, where ethylene flows through the inner pipe and the primary air flows through the outer pipe in a swirling motion imposed by an axial swirler, resulting in intense turbulent mixing inside the burning chamber. The reactive Navier-Stokes equations in the formulation of low-Mach number regime and Nek5000’s reactive plug-in was employed to calculate the thermal and transport properties along the chemical source terms. A 62-species reduced version of the ABF mechanism was utilized as the chemical mechanism in order to incorporate large soot PAH precursors up to A4.

Speaker: Dimitris Papageorgiou (Aristotle University of Thessaloniki)

2:20 PM → 2:40 PM Dispersed Microbubble-Laden Turbulent Flow Based on High-Order Euler-Lagrange Approach

To resolve multiphase flow, specifically dispersed phase flow, tracking the dispersed phase's trajectory is crucial. The Euler-Lagrange approach is adopted to predict the interaction between the dispersed phase (microbubble) and the continuous phase (turbulence). The Lagrangian tracking code, ppiclF (parallel particle-in-cell library written in Fortran), and the spectral element method code, Nek5000, are combined to simulate microbubble-laden turbulent flows. In this presentation, the developed microbubble models and microbubble dynamics in the turbulent channel flow will be introduced. Turbulent quantities such as turbulent boundary layer and Reynolds stresses are compared with respect to the size and number of bubbles. Additionally the drag reduction mechanism by microbubbles is analyzed.

Speaker: Byeong-Cheon Kim (University of Ulsan)

2:40 PM → 3:00 PM Flow Characteristics in Two-Dimensional and Three-Dimensional Thermo-Diffusive Unstable Flames

Hydrogen-based fuel burning faces challenges due to intrinsic premixed flame instabilities that affect flame morphology. This study introduces a three-dimensional (3D) Direct Numerical Simulation (DNS) dataset using a low-Mach formulation and a deficient reactant thermochemical model with NekRS, comparing flame characteristics to a well-established two-dimensional (2D) dataset [1].

Comparing the 2D and 3D flames reveals similar finger-like structures, but 3D flames exhibit 10% higher superadiabatic temperatures at cusps due to higher local curvatures enhancing local differential diffusion effects. The 3D flames also demonstrate higher mean consumption speeds, with the smallest 3D flame being faster than all 2D cases and 2.5 times its 2D counterpart's speed. The 3D flames exhibit a narrower range of flame curvature but higher displacement speeds and tangential strain rates. Overall, 3D flames show more positive stretch, leading to higher consumption speeds and reaction rates.

The study concludes that 3D flames exhibit higher temperature peaks and faster reactions compared to 2D flames, attributed to increased thermal diffusion and more positive flame stretch characteristics in 3D.

Reference
[1] Creta, Francesco, et al. Combustion and Flame 216 (2020): 256-270.

Speaker: Hamid Kavari (Sapienza University of Rome)

3:00 PM → 3:30 PM Coffee Break

3:30 PM → 3:50 PM Topology Optimization of Roughness Elements in Boundary Layers

This article applies density-based topology optimization in order to design roughness elements capable of generating stable streaks to damp the growth of Tollmien-Schlichting (TS) waves in a boundary layer. First a steady baseflow is established, then the unsteady linearized Navier-Stokes equations are evolved to assess the spatial growth of the TS waves across the flat plate. The optimization procedure aims to minimize the TS wave amplitude at a given downstream position while a novel constraint is used promoting a stable baseflow. This method has been applied to three initial material distributions yielding three distinct and novel designs capable of damping the downstream growth of the TS wave significantly more than a reference Minature Vortex Generator (MVG) of comparable size. The optimized designs and streaky baseflows they induce are then studied a posteriori using an energy budget analysis and local stability analysis.

Speaker: Harrison Nobis (KTH)

3:50 PM → 4:10 PM Ongoing NekRS Documentation Efforts

Speaker: Jez Swann

4:10 PM → 4:20 PM Closing

Speaker: Mathis Bode (Forschungszentrum Jülich GmbH)

6:00 PM → 8:00 PM Social Event

Wed, July 31

9:00 AM → 9:10 AM Opening

Speaker: Mathis Bode (Forschungszentrum Jülich GmbH)

9:10 AM → 10:30 AM Hands-on

Speaker: Jez Swann

10:30 AM → 11:00 AM Coffee Break

11:00 AM → 12:20 PM Hands-on

Speaker: Jez Swann

12:20 PM → 12:30 PM Closing

Speaker: Mathis Bode (Forschungszentrum Jülich GmbH)