Using ARM Performance Reports

ARM Performance Reports is a tool to get a high level understanding of the CPU, Memory and IO usage of your code, thus characterizing its bounds.
The only mandatory requirement for a code to be profiled with performance report tool is that it is dynamically linked. ARM Performance Report is installed on both Shaheen and Ibex.

perf-report on Ibex

To run perf-report on Ibex, you can do so in a SLURM jobscript. Here we are profiling a C code build from source using OpenMPI and dynamically linked. The executable heat_hybrid is a MPI+OpenMP code which we wish to run with 4 MPI tasks and 8 OpenMP threads per MPI task.

#!/bin/bash #SBATCH -n 4 #SBATCH -c 8 #SBATCH -N 1 #SBATCH -t 00:10:00 #SBATCH -A ibex-cs module load openmpi/4.0.3 module load arm-forge/20.2.1 export ALLINEA_LICENSE_FILE=$PERF_REPORTS_LICENCE_FILE export OMP_NUM_THREADS=8 perf-report --mpi -n 4 --openmp-threads=${OMP_NUM_THREADS} ./heat_hybrid


The code, when complete, produces two files, a text file and an HTML file, as reports.

Command: /ibex/scratch/shaima0d/scratch/forge_test/advanced-parallel-prog/hybrid/heat-fine/c/solution/heat_hybrid Resources: 1 node (128 physical, 128 logical cores per node) Memory: 504 GiB per node Tasks: 4 processes, OMP_NUM_THREADS was 8 Machine: cn514-05-r Start time: Tue Jan 11 14:27:12 2022 Total time: 3 seconds Full path: /ibex/scratch/shaima0d/scratch/forge_test/advanced-parallel-prog/hybrid/heat-fine/c/solution Summary: heat_hybrid is MPI-bound in this configuration Compute: 29.6% |==| MPI: 70.4% |======| I/O: 0.0% | This application run was MPI-bound. A breakdown of this time and advice for investigating further is in the MPI section below. CPU: A breakdown of the 29.6% CPU time: Single-core code: 9.5% || OpenMP regions: 90.5% |========| Scalar numeric ops: 1.5% || Vector numeric ops: 31.4% |==| Memory accesses: 45.1% |====| The per-core performance is memory-bound. Use a profiler to identify time-consuming loops and check their cache performance. MPI: A breakdown of the 70.4% MPI time: Time in collective calls: 6.3% || Time in point-to-point calls: 93.7% |========| Effective process collective rate: 0.00 bytes/s Effective process point-to-point rate: 26.8 MB/s I/O: A breakdown of the 0.0% I/O time: Time in reads: 0.0% | Time in writes: 0.0% | Effective process read rate: 0.00 bytes/s Effective process write rate: 0.00 bytes/s No time is spent in I/O operations. There's nothing to optimize here! OpenMP: A breakdown of the 90.5% time in OpenMP regions: Computation: 95.8% |=========| Synchronization: 4.2% || Physical core utilization: 25.0% |==| System load: 80.6% |=======| Physical core utilization is low and some cores may be unused. Try increasing OMP_NUM_THREADS to improve performance. Memory: Per-process memory usage may also affect scaling: Mean process memory usage: 166 MiB Peak process memory usage: 208 MiB Peak node memory usage: 24.0% |=| The peak node memory usage is very low. Running with fewer MPI processes and more data on each process may be more efficient. Energy: A breakdown of how energy was used: CPU: not supported System: not supported Mean node power: not supported Peak node power: 0.00 W Energy metrics are not available on this system. CPU metrics are not supported (no intel_rapl module)


The report above provides a high level breakdown of where the CPU time went and how much memory was used. It also reports on the fraction of time spent in OpenMP regions and how well the cores were utilized. The adviser provides good suggestions which can improve the performance of subsequent runs.