...
dcp
or distributed copy is a MPI-based copy tool developed by Lawrance Livermore National Lab (LLNL) as part of their mpifileutils
suite. We have installed it on Shaheen. Here is an example jobscript to launch a data moving job with dcp
:
Code Block | ||
---|---|---|
| ||
#!/bin/bash #SBATCH --ntasks=4 #SBATCH --time=01:00:00 #SBATCH --hint=nomultithread module load mpifileutils time srun -n ${SLURM_NTASKS} dcp --verbose --progress 60 --preserve /path/to/source/directory /path/to/destination/directory |
...
The table below compares the baseline time taken by cp
command to copy these files from /project
to /scratch
with that taken by dcp
with different number of MPI processes:
(MPI) processes | Time to completion | Speedup | |
---|---|---|---|
| 1 (serial) | 1139.75 seconds | 1 |
| 4 | 888.966 seconds | 1.282 |
| 16 | 226.064 seconds | 5.042 |
| 32 | 401.479 seconds | 2.838 |
Some observations
Given large enough number of files, you can see decent gains in using
dcp
. Use>1
Lustre strip when writing big files. This will increase throughput on each file sincedcp
does not decompose the files into blocks itself.Throwing more MPI processes may not always give you the more speedup, as seen in the case of 32 vs 16 MPI processes in the table above. Significantly less work (i.e. files to copy) per MPI process can introduce the MPI overhead of synchronizing and slows down the whole job. Thus there is a sweet spot.
It is possible to have variability in time give the time when your copy job runs and the load on the metadata server of Lustre filesystem.