OpenFOAM v2006

Owned by Mohsin Ahmed Shaikh
Last updated: Apr 12, 2022

OpenCFD’s OpenFOAM stable release from June 2020.

It has exposed ways to cleanly point to pre-installed dependency libraries on Shaheen.

ssh cdl5

Setup environment

module swap PrgEnv-cray PrgEnv-gnu module load cmake #For Scotch, metis, and HYPRE module load cray-tpsl #For petsc module load cray-petsc-64 #For Boost and CGAL module load boost cgal #For FFTW module load cray-fftw

Since ADIOS2 is used, I will let OpenFOAM’s installer build it from its Third-party software tarball.

Changes made to configuration files

Change/set the following in OpenFOAM-v2006/etc/bashrc

export WM_COMPILER_TYPE=system export WM_COMPILER=Cray export WM_PRECISION_OPTION=DP # Do a 32 bit double build otherwise libscotch will fail with domain decomposition. Only supported with 32 bit. export WM_LABEL_SIZE=32 export WM_COMPILE_OPTION=Opt export WM_MPLIB=CRAY-MPICH export WM_NCOMPPROCS=8

Update the choice of MPI library in bashrc:

export WM_MPLIB=CRAY-MPICH

Now update dependency libraries:

cd OpenFOAM-v2006/etc/config.sh

Change in file: scotch

SCOTCH_VERSION=scotch-system export SCOTCH_ARCH_PATH=$CRAY_TPSL_PREFIX_DIR

Change in file: metis

METIS_VERSION=metis-system export METIS_ARCH_PATH=$CRAY_TPSL_PREFIX_DIR

Change in file: hypre

hypre_version=hypre-system export HYPRE_ARCH_PATH=$CRAY_TPSL_PREFIX_DIR

Change in file: petsc

petsc_version=petsc-system export PETSC_ARCH_PATH=$PETSC_DIR

Change in file: CGAL

boost_version=boost-system cgal_version=cgal-system export BOOST_ARCH_PATH=${BOOST_ROOT}/../ export CGAL_ARCH_PATH=$CGAL_ARCH_PATH

Change in file: FFTW

fftw_version=fftw-system export FFTW_ARCH_PATH=$FFTW_ROOT

Change in file: cmake

export CMAKE_ARCH_PATH=$CMAKE_DIR

Change in file: ThirdParty-v2006/makeAdios2 [Lines: 155,156]

CC=cc CXX=CC

wmake general rules for linux64Cray needs updating

cd OpenFOAM-v2006/wmake/rules/linux64Cray cp ../General/mplibCRAY-MPICH . mv mplibINTELMPI .mplibINTELMPI

Following modifications to wmake/rules/General/mplibMPICH and wmake/rules/linux64Cray/mplibCRAY-MPICH got rid of the erroneous inclusion of -lmpi to the link line.

wmake/rules/General/mplibMPICH:

PFLAGS = -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX PINC = -isystem $(MPI_ARCH_PATH)/include PLIBS = -L$(MPI_ARCH_PATH)/lib$(WM_COMPILER_LIB_ARCH) -L$(MPI_ARCH_PATH)/lib -lmpich -lmpichcxx -lrt

wmake/rules/lib64Cray/mplibCRAY-MPICH:

include $(GENERAL_RULES)/mplibMPICH PFLAGS = -DOMPI_SKIP_MPICXX PINC = -I$(MPI_ARCH_PATH)/include PLIBS =

Change mpirun to srun in discovery of mpirun path in the file OpenFOAM-v2006/wmake/scripts/list_tools

echo "mpirun=$(command -v srun || true)"

Setup environment for compilation

source ./etc/bashrc
./Allwmake -l -a

In case a clean re-build is needed, the following can be run before Allwmake

wclean -a