OpenFOAM v2006

OpenCFD’s OpenFOAM stable release from June 2020.

It has exposed ways to cleanly point to pre-installed dependency libraries on Shaheen.

ssh cdl5

Setup environment

module swap PrgEnv-cray PrgEnv-gnu
module load cmake
#For Scotch, metis, and HYPRE
module load cray-tpsl
#For petsc
module load cray-petsc-64
#For Boost and CGAL
module load boost  cgal
#For FFTW
module load cray-fftw

Since ADIOS2 is used, I will let OpenFOAM’s installer build it from its Third-party software tarball.

Changes made to configuration files

Change/set the following in OpenFOAM-v2006/etc/bashrc

export WM_COMPILER_TYPE=system
export WM_COMPILER=Cray
export WM_PRECISION_OPTION=DP
# Do a 32 bit double build otherwise libscotch will fail with domain decomposition. Only supported with 32 bit.
export WM_LABEL_SIZE=32
export WM_COMPILE_OPTION=Opt
export WM_MPLIB=CRAY-MPICH
export WM_NCOMPPROCS=8

Update the choice of MPI library in bashrc:

Now update dependency libraries:

Change in file: scotch

Change in file: metis

Change in file: hypre

Change in file: petsc

Change in file: CGAL

Change in file: FFTW

Change in file: cmake

Change in file: ThirdParty-v2006/makeAdios2 [Lines: 155,156]

wmake general rules for linux64Cray needs updating

Following modifications to wmake/rules/General/mplibMPICH and wmake/rules/linux64Cray/mplibCRAY-MPICH got rid of the erroneous inclusion of -lmpi to the link line.

wmake/rules/General/mplibMPICH:

wmake/rules/lib64Cray/mplibCRAY-MPICH:

Change mpirun to srun in discovery of mpirun path in the file OpenFOAM-v2006/wmake/scripts/list_tools

Setup environment for compilation

In case a clean re-build is needed, the following can be run before Allwmake