memP is a Lawrence Livermore National Labs (LLNL) developed, light weight, parallel heap profiling library. Its primarily designed to identify the heap allocation that causes an MPI task to reach its memory in use high water mark (HWM).
Summary Report: Generated from within MPI_Finalize, this report describes the memory HWM of each task over the run of the application. This can be used to determine which task allocates the most memory and how this compares to the memory of other tasks.
Task Report: Based on specific criteria, a report can be generated for each task, that provides a snapshot of the heap memory currently in use, including the amount allocated at specific call sites.
Load the memp Module
module load memP
Compile with the recommended BG/Q flags and link your application with the required libraries:
mpixlc -g -Wl,-zmuldefs -o myprog myprog.c -L/usr/local/tools/memP/lib -lmemP mpixlf77 -g -Wl,-zmuldefs -o myprog myprog.f -L/usr/local/tools/memP/lib -lmemP
Then run your MPI application as usual and you will see an memP header and traile it sends to stdout, as well as the output file generated at the end of the run.
See http://memp.sourceforge.net/ for full details.