Installing personalized Python modules

On Shaheen you will find several python flavors, namely, inteppython3, python/3.8.0, and cray-python. All these come with most of the popular non-standard python packages installed in addition to the core packages.

If however, you see any module missing, you can install in your allocated space on shared filesystem and update runtime environment for the installed newly python package.

Installation using pip package manager

On Shaheen, we recommend you install in your project directory /project/your-project-id/your-user-id/software. On Ibex you may install these packages in /scratch/your-user-id/software.

Most Python packages come with a pip installation recipe, i.e. they are hosted on popular Pypi channel or elsewhere, eg. GitHub etc. Here is how you may install and point to the desired installation directory. For the example below I am using intelpython3 as my base python and installing e.g. tensorboard, a popular web application for monitoring performance metrics of a deep learning training job.

1 2 module load intelpython3 pip install --prefix=/project/your-project-id/your-user-id/software tensorboard==2.5.0

The above will install tensorboard package in my project directory e.g. /project/k01/shaima0d/software/lib/python3.7/site-pacakges, where k01 is my project ID and shaima0d is my User ID respective (you should use your own).

Using the installed package

You can inform the python in your environment to search for python packages to an additional path including the preset default paths by setting an environment variable PYTHONPATH. Precedence applies.

1 export PYTHONPATH=/project/your-project-id/your-user-id/software/lib/python3.7/site-pacakges:$PYTHONPATH

The above line must be exported before invoking python either interactively or as a batch script in your jobscripts, e.g. (on Shaheen)

1 2 3 4 5 6 7 8 9 #!/bin/bash #SBATCH --time=01:00:00 #SBATCH --nodes=1 #SBATCH --hint=nomultithread module load intelpython3 export PYTHONPATH=/project/k01/shaima0d/software/lib/python3.7/site-pacakges:$PYTHONPATH python train.py

 

If there are warnings or error of not having right permissions to upgrade a dependency package, contact us and our applications team will assist you with installation.