Motivation

Sometimes it is needed to copy large number of files from /scratch to /project or vice versa. Both cp and rsync are convenient but sometimes you need speed to do such task.

Distributed Copy

dcp or distributed copy is a MPI-based copy tool developed by Lawrance Livermore National Lab (LLNL) as part of their mpifileutils suite. We have installed it on Shaheen. Here is an example jobscript to launch a data moving job with dcp:

#!/bin/bash 

#SBATCH --ntasks=4
#SBATCH --time=01:00:00
#SBATCH --hint=nomultithread

module load mpifileutils
time srun -n ${SLURM_NTASKS} dcp --verbose --progress 60 --preserve /path/to/source/directory /path/to/destination/directory

...

The table below compares the baseline time taken by cp command to copy these files from /project to /scratch with that taken by dcp with different number of MPI processes:

Some observations

• Given large enough number of files, you can see decent gains in using dcp. Use >1 Lustre strip when writing big files. This will increase throughput on each file since dcp does not decompose the files into blocks itself.

• Throwing more MPI processes may not always give you the more speedup, as seen in the case of 32 vs 16 MPI processes in the table above. Significantly less work (i.e. files to copy) per MPI process can introduce the MPI overhead of synchronizing and slows down the whole job. Thus there is a sweet spot.

• It is possible to have variability in time give the time when your copy job runs and the load on the metadata server of Lustre filesystem.