Using TAU on surveyor¶
Start by reading the main profiling page
Do not using the customize.py from the above page, following the instructions found on this page instead. The following mostly applies to automatic instrumentation using the TAU compiler scripts.
Overhead with the TAU for automatic instrumentation has been measured at about 20% for the b256H2O.py. Use manual instrumentation if this overhead is unacceptable or you might need to add more functions to the existing selective instrumentation file select.tau
The version of TAU at ALCF is updated frequently, please check: https://wiki.alcf.anl.gov/index.php/Tuning_and_Analysis_Utilities_(TAU)
We first described the case of automatically instrumentation with the GCC and XLC compilers.
The instructions below are for using source-based automatic instrumentation with TAU. It is well tested with the GCC compiler. It should also work with XLC compiler, but is untested.
Add the following lines to you
TAUARCHITECTURE = bgp TAUVERSION = 2.19.2 TAU_MAKEFILE = /soft/apps/tau/tau-$TAUVERSION/bgp/lib/Makefile.tau-bgptimers-gnu-mpi-python-pdt TAU_OPTIONS = '-optVerbose -optShared -optTauSelectFile=select.tau \ -optTau="-rn Py_RETURN_NONE -i/soft/apps/tau/tau-'$TAUVERSION'/include/TAU_PYTHON_FIX.h"' PATH += /soft/apps/tau/tau-$TAUVERSION/$TAUARCHITECTURE/bin
The bindings are located in
/soft/apps/tau/tau/tau-$TAUVERSION/bgp/lib/<name>. This particular TAU library binding supports BGP timers (a low-level
timer with minimal overhead), MPI, GNU compiler, and Python. This is the recommended library binding for flat profiles.
You will need to change one line in your bgp_gcc.py:
cmd = "tau_cc.sh %s %s"%(flags, cmd)
It is recommend that dynamic linking be used for the MPI libraries instead of static linking, see the comments in configure_surveyor.py
Additionally, the TAU library must be linked manually. See the comments in customize_surveyor.py
It is straightforward to swap the default Python timers with those provided by TAU and make subclass of the default GPAW calculator:
from gpaw.utilities.timing import TAUTimer class MyGPAW(GPAW): timer_class = TAUTimer calc = MyGPAW(...)
Run time environment variables¶
Here are the recommended run time environment variables that should be passed to Cobalt via qsub:
TAU_COMPENSATE seems to cause problems with manual instrumentation, so do not set it to 0 which means off. In any case, it should not be particularly relevant unless you have manual timers on a frequently accessed lightweight functions.
Modification to the submission script for use with TAU are show in the example submission script surveyor.sh
If you are doing manual instrumentation, simply pass the actual input file to
gpaw-python instead. For automatic instrumentation, you need to
import tau def OurMain(): import CH4; tau.run('OurMain()')
TAU run will then produce
profile.* files that can be merged into
the default TAU’s
ppk format using the command issued from the directory
profile.* files reside:
paraprof --pack CH4.ppk
The actual analysis can be made on a different machine, by transferring
CH4.ppk file from
surveyor, installing TAU, and launching: