Running CDSAPI on Shaheen as container job
With changes in Shaheen programming environment, it is sometimes necessary to upgrade python and its dependencies. This implies that all the packages depending on python will also need updating. Conda environment are good up an alternative package management strategy which freezes the runtime environment is containers. This container image should work on Shaheen, Ibex or Neser.
Running rootless containers on Shaheen is easy. Here is an example of cdsapi
container which KSL team has created and uploaded in the DockerHub registry for our users.
Pull the Image
Before you run cdsapi
as a container job Shaheen, first pull the image on Shaheen filesystem:
cd /scratch/$USER
module load singularity
singularity pull docker://krccl/cdsapi:0.5.1
The above should produce a file named cdsapi_0.5.1.sif
CDSAPI key file
Create an API Key file needed by cdsapi
as you identity for the remote repositories to recognise your account. The API key can be generated by creating an account using the following instruction: https://cds.climate.copernicus.eu/api-how-to
Copy the url
and API key
and paste in the file /scratch/$USER/cdsapi_rc
Running container job on compute node as batch job
Here is a jobscript that runs cdsapi
as a batch job on Shaheen compute node:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --time=00:10:00
module load singularity
export IMAGE=./cdsapi_0.5.1.sif
export SINGULARITYENV_CDSAPI_RC=$PWD/cdsapi_key
srun singularity exec $IMAGE python test.py
Where test.py
looks as follows:
import cdsapi
c = cdsapi.Client()
c.retrieve("reanalysis-era5-pressure-levels",
{
"variable": "temperature",
"pressure_level": "1000",
"product_type": "reanalysis",
"year": "2008",
"month": "01",
"day": "01",
"time": "12:00",
"format": "grib"
}, "download.grib")
Running container job on compute node as interactive job
Similar to the batch job as above, you can run the contents of test.py
in your python interpreter within a container:
First allocate an interactive job
Then jump on the compute node allocated to you:
Load the environment
Run Singularity shell
The above will put you in a container environment where the Python and CDSAPI are available. You can run your python script interactively step by step:
To exit the container
Please feel free to comment on the page.
Happy hunting