OpenFOAM v2006
OpenCFD’s OpenFOAM stable release from June 2020.
It has exposed ways to cleanly point to pre-installed dependency libraries on Shaheen.
ssh cdl5
Setup environment
module swap PrgEnv-cray PrgEnv-gnu
module load cmake
#For Scotch, metis, and HYPRE
module load cray-tpsl
#For petsc
module load cray-petsc-64
#For Boost and CGAL
module load boost cgal
#For FFTW
module load cray-fftw
Since ADIOS2 is used, I will let OpenFOAM’s installer build it from its Third-party software tarball.
Changes made to configuration files
Change/set the following in OpenFOAM-v2006/etc/bashrc
export WM_COMPILER_TYPE=system
export WM_COMPILER=Cray
export WM_PRECISION_OPTION=DP
# Do a 32 bit double build otherwise libscotch will fail with domain decomposition. Only supported with 32 bit.
export WM_LABEL_SIZE=32
export WM_COMPILE_OPTION=Opt
export WM_MPLIB=CRAY-MPICH
export WM_NCOMPPROCS=8
Update the choice of MPI library in bashrc:
export WM_MPLIB=CRAY-MPICH
Now update dependency libraries:
cd OpenFOAM-v2006/etc/config.sh
Change in file: scotch
SCOTCH_VERSION=scotch-system
export SCOTCH_ARCH_PATH=$CRAY_TPSL_PREFIX_DIR
Change in file: metis
METIS_VERSION=metis-system
export METIS_ARCH_PATH=$CRAY_TPSL_PREFIX_DIR
Change in file: hypre
hypre_version=hypre-system
export HYPRE_ARCH_PATH=$CRAY_TPSL_PREFIX_DIR
Change in file: petsc
petsc_version=petsc-system
export PETSC_ARCH_PATH=$PETSC_DIR
Change in file: CGAL
boost_version=boost-system
cgal_version=cgal-system
export BOOST_ARCH_PATH=${BOOST_ROOT}/../
export CGAL_ARCH_PATH=$CGAL_ARCH_PATH
Change in file: FFTW
fftw_version=fftw-system
export FFTW_ARCH_PATH=$FFTW_ROOT
Change in file: cmake
export CMAKE_ARCH_PATH=$CMAKE_DIR
Change in file: ThirdParty-v2006/makeAdios2
[Lines: 155,156]
CC=cc
CXX=CC
wmake
general rules for linux64Cray
needs updating
cd OpenFOAM-v2006/wmake/rules/linux64Cray
cp ../General/mplibCRAY-MPICH .
mv mplibINTELMPI .mplibINTELMPI
Following modifications to wmake/rules/General/mplibMPICH
and wmake/rules/linux64Cray/mplibCRAY-MPICH
got rid of the erroneous inclusion of -lmpi
to the link line.
wmake/rules/General/mplibMPICH
:
PFLAGS = -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX
PINC = -isystem $(MPI_ARCH_PATH)/include
PLIBS = -L$(MPI_ARCH_PATH)/lib$(WM_COMPILER_LIB_ARCH) -L$(MPI_ARCH_PATH)/lib -lmpich -lmpichcxx -lrt
wmake/rules/lib64Cray/mplibCRAY-MPICH
:
include $(GENERAL_RULES)/mplibMPICH
PFLAGS = -DOMPI_SKIP_MPICXX
PINC = -I$(MPI_ARCH_PATH)/include
PLIBS =
Change mpirun
to srun
in discovery of mpirun path in the file OpenFOAM-v2006/wmake/scripts/list_tools
echo "mpirun=$(command -v srun || true)"
Setup environment for compilation
source ./etc/bashrc
./Allwmake -l -a
In case a clean re-build is needed, the following can be run before Allwmake
wclean -a