This page describes how to use the Climate Data Operators (CDO) software installed in a container. The CDO version in use here is 1.9.10 but similar can be created for other versions and for other tools pre-postprocessing tools from weather and climate model domain.
Generally the containerization of any workload or application is motivated by either by its complexity to install or by its usage pattern which may require its availability on multiple platforms which vary in infrastructure e.g. Operating system, scientific libraries and their versions. Containerizing an application also makes it immune to maintenance cycles the compute platforms go through, such is the case with Shaheen, Neser and Ibex.
An image refers to an artifact which contains all the workload along with all the dependencies and Operating system of choices ( minus the kernel). From this image, a container runtime can be used to create a container, and run your command along with its arguments.
cdo/1.9.10 is available on KSL’s public registry of images in hub.docker.com.
Shaheen, Neser and Ibex commonly use a container platform called singularity.
On Neser, you can do the following to pull the CDO image:
cd $HOME && mkdir -p tmpdir export SINGULARITY_TMPDIR=$HOME/tmpdir singularity pull docker://krccl/cdo_gnu:1.9.10 |
This may take some time to finish and upon successful completion, you will have file called cdo_gnu_1.9.10.sif . This is a binary file in singularity's Image Format.
You can copy this image to your scratch directory so it is accessible on compute nodes:
cp ~/cdo_gnu_1.9.10.sif /scratch/$USER/images/ |
To run, you simply prepend your cdo command with the necessary singularity prefix.
singularity run /scratch/$USER/images/cdo_gnu_1.9.10.sif cdo --version |
The output is as expected:
Climate Data Operators version 1.9.10 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -g -O2 -fopenmp -pthread
CXX version : g++ (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
C Compiler: gcc -g -O2 -fopenmp -pthread -pthread
C version : gcc (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
Features: 250GB 80threads C++14 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5 OPeNDAP PROJ XML2 MAGICS CURL SSE2
Libraries: HDF5/1.12.2 proj/5.0.1 xml2/2.9.4 curl/7.58.0
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.10
cgribex library version : 1.9.5
ecCodes library version : 2.26.0
NetCDF library version : 4.9.0 of Aug 17 2022 09:14:20 $
hdf5 library version : 1.12.2
exse library version : 1.4.2
FILE library version : 1.9.1 |
Please feel free to email help@hpc.kaust.edu.sa for requesting supporting regarding the image or containerized runs of CDO on any KSL platform.
Please feel free to email help@hpc.kaust.edu.sa for requesting supporting regarding the image or containerized runs of CDO on any KSL platform.