In some situations you may want to look at the real time utilization of the compute resources allocated to you. There are multiple way of doing this. NVIDIA published a neat solution which visualizes the time series output from nvidia-smi in a Bokeh dashboard. nvidia-smi uses nvml library to collect the metrics.

To install in your conda environment you can use pip:

pip install jupyterlab-nvdashboard

You can then launch the dashboard server in your jobscript:

#!/bin/bash
#SBATCH --gpus=1
#SBATCH --time=00:10:00

# Try different port if the following is occupied by another user
nvdashboard 10101 &
echo "ssh -L 10101:$(/bin/hostname):10101 $USER@glogin.ibex.kaust.edu.sa"
sleep 10

python train.py

To connect to this server, you will first establish ssh tunnel to the compute node it your training job is running on. For example the squeue -u $USER command tells you that your job has started on gpu212-04 node. Open a new terminal and run the following command: 

ssh -L 10101:gpu212-04:10101 username@glogin.ibex.kaust.edu.sa
note

Please use your username in place in the above command

Please use your username in place in the above command

Once logged in, use the following URL in your browser to access the dashboard:

http://localhost:10101

The following dashboard will show:

You can select the metrics of your choice. For instance if I wish to see all the things related to GPUs:

Few important things to note