jugene.fz-juelich.de

Here you find information about the system http://www.fz-juelich.de/jsc/jugene.

Numpy needs to be build with powerpc-bgp-linux-gfortran instead of gfortran compiler, so in order to build numpy specify the environment variable F90:

$ export F90=/bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/bin/gfortran

After that, numpy can be installed to $HOME/python as:

$ ldpath=/bgsys/drivers/ppcfloor/gnu-linux/lib
$ p=/bgsys/drivers/ppcfloor/gnu-linux/bin/python
$ LD_LIBRARY_PATH="$ldpath" $p setup.py install --home=$HOME/python

In order to build GPAW, use the following customize.py:

scalapack = True

libraries = [
           'scalapack',
           'blacsCinit',
           'blacsF77init',
           'blacs',
           'lapack',
           'esslbg',
            'xl',
            'xlopt',
            'xlf90_r',
            'xlfmath',
            'pthread',
            'xlomp_ser',
           ]

library_dirs = [
          '/bgsys/local/lib/',
           '/opt/ibmcmp/xlf/bg/11.1/lib',
           '/opt/ibmcmp/xlsmp/bg/1.7/lib',
           '/bgsys/drivers/ppcfloor/gnu-linux/lib'
           ]

extra_compile_args += ['-std=c99']

define_macros += [('GPAW_AIX', '1')]
define_macros += [('GPAW_BGP', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_BLAS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_LAPACK', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_BLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_SCALAPACK', '1')]

Because of missing popen3 function you need to remove all the contents of the gpaw/version.py file after version = '0.4'. The same holds for ase/version.py in the ase installation! Suggestions how to skip the popen3 testing in gpaw/version.py on BGP are welcome!

Here is an example of batch job script:

 #!/bin/bash
# @ job_name = hello
# @ output = $(job_name).o$(jobid)
# @ error = $(job_name).e$(jobid)
# @ wall_clock_limit = 00:12:00
# @ notification = never
# @ notify_user = my_email@csc.fi
# @ job_type = bluegene
# @ bg_size = 1
# @ queue
home=/homea/prace/prace025
prog=${home}/python/bin/gpaw-python
args="${home}/test-gpaw/CH4.py"

mpirun=/bgsys/drivers/ppcfloor/bin/mpirun

ldpath="/bgsys/local/lib/ibmcmp/lib/bglib"

pythonpath="${home}/python/lib/python/"

gpaw_setups="${home}/gpaw-setups-0.4.2039"

runargs="-np 4"

runargs="$runargs -cwd $PWD"
runargs="$runargs -mode SMP"
runargs="$runargs -env LD_LIBRARY_PATH=$ldpath -env PYTHONPATH=$pythonpath -envGPAW_SETUP_PATH=$gpaw_setups"

echo "Hello. This is `hostname` at `date` `pwd`"

echo "$mpirun $runargs -exe $prog $args"
/usr/bin/time $mpirun $runargs -exe $prog -args $args

echo "Program completed at `date` with exit code $?."

The batch jobs are submitted with llsubmit:

$ llsubmit job_file