Installing personalized Python modules
On Shaheen you will find several python flavors, namely, inteppython3, python/3.8.0, and cray-python. All these come with most of the popular non-standard python packages installed in addition to the core packages.
If however, you see any module missing, you can install in your allocated space on shared filesystem and update runtime environment for the installed newly python package.
Installation using pip package manager
On Shaheen, we recommend you install in your project directory /project/your-project-id/your-user-id/software. On Ibex you may install these packages in /scratch/your-user-id/software.
Most Python packages come with a pip installation recipe, i.e. they are hosted on popular Pypi channel or elsewhere, eg. GitHub etc. Here is how you may install and point to the desired installation directory. For the example below I am using intelpython3 as my base python and installing e.g. tensorboard, a popular web application for monitoring performance metrics of a deep learning training job.
module load intelpython3
pip install --prefix=/project/your-project-id/your-user-id/software tensorboard==2.5.0The above will install tensorboard package in my project directory e.g. /project/k01/shaima0d/software/lib/python3.7/site-pacakges, where k01 is my project ID and shaima0d is my User ID respective (you should use your own).
Using the installed package
You can inform the python in your environment to search for python packages to an additional path including the preset default paths by setting an environment variable PYTHONPATH. Precedence applies.
export PYTHONPATH=/project/your-project-id/your-user-id/software/lib/python3.7/site-pacakges:$PYTHONPATHThe above line must be exported before invoking python either interactively or as a batch script in your jobscripts, e.g. (on Shaheen)
#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --hint=nomultithread
module load intelpython3
export PYTHONPATH=/project/k01/shaima0d/software/lib/python3.7/site-pacakges:$PYTHONPATH
python train.py
If there are warnings or error of not having right permissions to upgrade a dependency package, contact us and our applications team will assist you with installation.