KAUST Supercomputing Lab maintains a docker image with Horovod/0.19.2. If you wish to modify the image, here is the Dockerfile you can use to recreate an image with desired modification (download Mellanox OFED tarball MLNX_OFED_LINUX-5.0-126.96.36.199-ubuntu18.04-x86_64.tgz) On Ibex you can use this image to run a container with Singularity platform. Here is an example:
On the glogin node you can pull the image from DockerHub:
Once you end up pulling the image successfully, singularity will convert it into a Singularity Image File or SIF , which is a monolithic and static binary file (you can copy it in /ibex/scratch if you wish).
Here is an example Jobscript launching a horovod training job as singularity container:
Single node single GPU
You may possibly want to run a single GPU job for debugging: