CDO in a container

This page describes how to use the Climate Data Operators (CDO) software installed in a container. The CDO version in use here is 1.9.10 but similar can be created for other versions and for other tools pre-postprocessing tools from weather and climate model domain.

Motivation

Generally the containerization of any workload or application is motivated by either by its complexity to install or by its usage pattern which may require its availability on multiple platforms which vary in infrastructure e.g. Operating system, scientific libraries and their versions. Containerizing an application also makes it immune to maintenance cycles the compute platforms go through, such is the case with Shaheen, Neser and Ibex.

An image refers to an artifact which contains all the workload along with all the dependencies and Operating system of choices ( minus the kernel). From this image, a container runtime can be used to create a container, and run your command along with its arguments.

Getting access to the image

cdo/1.9.10 is available on KSL’s public registry of images in hub.docker.com.

Shaheen, Neser and Ibex commonly use a container platform called singularity.

On Neser, you can do the following to pull the CDO image:

cd $HOME && mkdir -p tmpdir export SINGULARITY_TMPDIR=$HOME/tmpdir singularity pull docker://krccl/cdo_gnu:1.9.10

This may take some time to finish and upon successful completion, you will have file called cdo_gnu_1.9.10.sif . This is a binary file in singularity's Image Format.

You can copy this image to your scratch directory so it is accessible on compute nodes:

cp ~/cdo_gnu_1.9.10.sif /scratch/$USER/images/

Running on any node (compute/login)

To run, you simply prepend your cdo command with the necessary singularity prefix.

singularity run /scratch/$USER/images/cdo_gnu_1.9.10.sif cdo --version

The output is as expected:

 

Please feel free to email help@hpc.kaust.edu.sa for requesting supporting regarding the image or containerized runs of CDO on any KSL platform.