Working with Conda environments on Shaheen 2

Owned by Mohsin Ahmed Shaikh
Last updated: Jun 11, 2023 by Kadir Akbudak

On Shaheen 2, /home is not mounted on compute nodes. This means, even though you can access yours files in your $HOME directory from login nodes but not from compute nodes. Another point to note is that as of January 2022, the /project directory is read-only from compute node. Both these changes have consequence if you are a conda user on Shaheen.

Here are some tips to keep the ball rolling and use conda environments on login and compute nodes on Shaheen.

In the documentation below, I am using my username and my project ID. Please replace is with your corresponding credentials in the paths where it applies.

Install Conda in project directory

You will need to install your miniconda in your project directory so it is accessible from reading on compute nodes. Please create a directory in by following the steps below in your project directory:

mkdir -p /project/k01/$USER/miniconda3 lfs setstripe -c 4 /project/k01/$USER/miniconda3

 

Here is me installing miniconda3 in my project directory as an example. Please use your Porject ID of choice in installation path:

cd /project/k01/$USER/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh -u Welcome to Miniconda3 py39_4.11.0 In order to continue the installation process, please review the license agreement. Please, press ENTER to continue >>> .... Notice of Third Party Software Licenses Do you accept the license terms? [yes|no] [no] >>> yes Miniconda3 will now be installed into this location: /home/shaima0d/miniconda3 - Press ENTER to confirm the location - Press CTRL-C to abort the installation - Or specify a different location below [/home/shaima0d/miniconda3] >>>/project/k01/shaima0d/miniconda3 ..... Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no] no You have chosen to not have conda modify your shell scripts at all. ..... Thank you for installing Miniconda3!

conda is now installed in our project directory.

Creating environments

For new environments, they should also be created in project directory so that they are accessible from compute nodes. To this end, you will need to set the following environment variable CONDA_PKGS_DIRS to direct the packages to be installed in a prescribed location other than the default which is $HOME/.cache

For example:

mkdir -p /scratch/shaima0d/conda_cache lfs setstripe -c 4 /scratch/shaima0d/conda_cache export CONDA_PKGS_DIRS=/scratch/shaima0d/conda_cache

To create a new environment you need to first initialize miniconda3 environment by sourcing it from the installation path and then create the new environment of your choice:

source /project/k01/shaima0d/miniconda3/bin/activate conda install mamba conda create -n tf26 -c intel tensorflow=2.6.0 conda activate tf26 pip install flatbuffers

Once the environment is created, let’s check where our Tensorflow has been installed.

Interactively using Conda environments

For testing interactively on a Shaheen compute node and launch the python interpreter freshly installed in our conda environment:

salloc -p debug srun --pty bash source /project/k01/shaima0d/miniconda3/bin/activate tf26 python
>>> import tensorflow as tf >>> tf.__file__ '/project/k01/shaima0d/miniconda3/envs/tf26/lib/python3.9/site-packages/tensorflow/__init__.py' >>> tf.__version__ '2.6.0'

Jobscript for batch processing

The following is an example of activating your conda environments in a SLRUM jobscript for batch processing: (tf_test.py is the target python script to run)

#!/bin/bash #SBATCH --ntasks=1 #SBATCH --time=00:10:00 source /project/k01/shaima0d/miniconda3/bin/activate tf26 srun -n 1 python tf_test.py

Steps involving sourcing the miniconda3 and activating the specific environment is suppose to be done every time you use an installed package. When installing a new package you should have the CONDA_PKGS_DIR set to your project directory.

 

Install environment in specific path

It is sometimes useful to relocate an environment and install it in a custom path/directory (.e.g in your scratch directory). For this it is suggested that you write an environment.yaml file to list all the software to install. Once done, the following will install the environment in a specific path:

mkdir -p /scratch/$USER/somewhere/my_env_name lfs setstripe -c 1 /scratch/$USER/somewhere/my_env_name conda env creaate -f environment.yaml -p /scratch/$USER/somewhere/my_env_name --force

Here is an example of environment.yaml :

name: my_env_name dependencies: - ipython=7.13 - matplotlib=3.1 - pandas=1.0 - pip=20.0 - python=3.6 - scikit-learn=0.22 - pip: - dask

 

To activate the above environment, you will source it as absolute path:

source /project/k01/$USER/miniconda3/bin/activate /scratch/$USER/somewhere/my_env_name