Clara-Parabricks on Ibex

Owned by Mohsin Ahmed Shaikh
Last updated: Jul 28, 2020

Installation

  • requires fakeroot feature of Singularity to be working

    • must build the image on a compute node

Getting the source

  • Goto https://developer.nvidia.com/clara-parabricks and choose appropriate Licensing method:

    • I chose “standard evaluation license” for testing

    • Fill in the requested details and the link to tarball containing the installation script and license will be emailed by NVIDIA

  • It appears that Parabricks is a container image, which when installed using the provided installer.py in the tarball, creates a Singularity sandbox as a installation directory and builds a statically built executable(s) pbrun.

Intall command

salloc --gres=gpu:1 --time=01:00:00

Once GPU node is allocated, the following needs to run on compute node:

export XDG_RUNTIME_DIR=/path/to/some/tmp/space srun --pty ./installer.py --container singularity --install-location /path/to/installation/directory

 

 

To pull a container image from NVIDIA GPU Containers Cloud, you will need to add --ngc to the installer command line. Before using the NGC container image please check the version, its v2.5.0 at the moment whereas the newest in gitlab in v3.0.0.2:

srun --pty ./installer.py --container singularity --ngc --install-location /path/to/installation

 

Testing

I am using the testing recipe from https://ngc.nvidia.com/catalog/containers/hpc:parabricks

Getting data

$ wget https://s3.amazonaws.com/parabricks.sample/parabricks_sample.tar.gz $ tar xf parabricks_sample.tar.gz $ ls parabricks_sample Data Ref

Typical Jobscript

#!/bin/bash #SBATCH --job-name=parabricks-test #SBATCH --time=02:00:00 #SBATCH --gres=gpu:1 #SBATCH --constraint=gpu_ai #SBATCH --ntasks=1 #SBATCH --cpus-per-task=4 #SBATCH --mem=20G module load parabricks/3.0.0.2 export DATA_DIR=/ibex/scratch/shaima0d/scratch/parabricks-testing pbrun fq2bam \ --ref ${DATA_DIR}/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq ${DATA_DIR}/parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \ --out-bam output.bam

Output

Loading module for Singularity Singularity 3.5 modules now loaded Loading module for parabricks-3.0.0.2 parabricks-3.0.0.2 modules now loaded [Parabricks Options Mesg]: Checking argument compatibility [Parabricks Options Mesg]: Automatically generating ID prefix [Parabricks Options Mesg]: Read group created for /ibex/scratch/shaima0d/scratch/parabricks- testing/parabricks_sample/Data/sample_1.fq.gz and /ibex/scratch/shaima0d/scratch/parabricks- testing/parabricks_sample/Data/sample_2.fq.gz [Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.0.0.2 || || GPU-BWA mem, Sorting Phase-I || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ [M::bwa_idx_load_from_disk] read 0 ALT contigs GPU-BWA mem ProgressMeter Reads Base Pairs Aligned [16:21:48] 5043564 580000000 [16:22:36] 10087128 1160000000 [16:23:25] 15130692 1740000000 [16:24:13] 20174256 2320000000 [16:25:01] 25217820 2900000000 [16:25:49] 30261384 3480000000 [16:26:38] 35304948 4060000000 [16:27:26] 40348512 4640000000 [16:28:16] 45392076 5220000000 [16:29:05] 50435640 5800000000 GPU-BWA Mem time: 525.787852 seconds GPU-BWA Mem is finished. GPU Sorting, Marking Dups, BQSR ProgressMeter SAM Entries Completed Total GPU-BWA Mem + Sorting + MarkingDups + BQSR Generation + BAM writing Processing time: 570.052968 seconds [main] CMD: PARABRICKS mem -Z ./pbOpts.txt /ibex/scratch/shaima0d/scratch/parabricks-testing/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /ibex/scratch/shaima0d/scratch/parabricks-testing/parabricks_sample/Data/sample_1.fq.gz /ibex/scratch/shaima0d/scratch/parabricks-testing/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 [main] Real time: 576.339 sec; CPU: 2121.508 sec ------------------------------------------------------------------------------ || Program: GPU-BWA mem, Sorting Phase-I || || Version: v3.0.0.2 || || Start Time: Mon Jul 27 16:20:45 2020 || || End Time: Mon Jul 27 16:30:21 2020 || || Total Time: 9 minutes 36 seconds || ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.0.0.2 || || Sorting Phase-II || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ progressMeter - Percentage [16:30:24] 0.0 0.00 GB [16:30:34] 47.6 0.27 GB [16:30:44] 94.0 0.11 GB Sorting and Marking: 30.002 seconds ------------------------------------------------------------------------------ || Program: Sorting Phase-II || || Version: v3.0.0.2 || || Start Time: Mon Jul 27 16:30:24 2020 || || End Time: Mon Jul 27 16:30:54 2020 || || Total Time: 30 seconds || ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.0.0.2 || || Marking Duplicates, BQSR || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ progressMeter - Percentage [16:31:05] 9.0 5.95 GB [16:31:15] 22.3 13.18 GB [16:31:25] 38.2 11.80 GB [16:31:35] 52.4 9.11 GB [16:31:45] 64.0 6.49 GB [16:31:55] 76.7 3.97 GB [16:32:05] 87.1 1.24 GB [16:32:15] 100.0 0.00 GB BQSR and writing final BAM: 80.932 seconds ------------------------------------------------------------------------------ || Program: Marking Duplicates, BQSR || || Version: v3.0.0.2 || || Start Time: Mon Jul 27 16:30:55 2020 || || End Time: Mon Jul 27 16:32:16 2020 || || Total Time: 1 minute 21 seconds || ------------------------------------------------------------------------------

Benchmarks

NOTES

  • Ran the benchmark with data in local SSD (/tmp of the compute node) with Single GPU. There was no change in compute time of phases.