uranus

The uranus machine is a cluster of dual socket, quad-core AMD Opteron 2354 CPUs, 2.2 GHz processors with 2 GB of memory per core.

Instructions assume bash, installation under ${HOME}/opt. Build the unoptimized numpy/scipy:

mkdir ${HOME}/opt
cd ${HOME}/opt
wget http://dfn.dl.sourceforge.net/sourceforge/numpy/numpy-1.3.0.tar.gz
wget http://dfn.dl.sourceforge.net/sourceforge/scipy/scipy-0.7.0.tar.gz
wget http://python-nose.googlecode.com/files/nose-0.11.0.tar.gz
tar zxf nose-0.11.0.tar.gz
tar zxf numpy-1.3.0.tar.gz
tar zxf scipy-0.7.0.tar.gz
cd nose-0.11.0
python setup.py install --prefix=${HOME}/opt/python 2>&1 | tee install.log
cd ../numpy-1.3.0
python setup.py install --prefix=${HOME}/opt/python 2>&1 | tee install.log
export PYTHONPATH=${HOME}/opt/python/lib64/python2.4/site-packages:${HOME}/opt/python/lib/python2.4/site-packages
cd ..
python -c "import numpy; numpy.test()"

wget http://www.netlib.org/blas/blas.tgz
tar zxf blas.tgz
export BLAS_SRC=${HOME}/opt/BLAS
wget http://www.netlib.org/lapack/lapack.tgz
tar zxf lapack.tgz
export LAPACK_SRC=${HOME}/opt/lapack-3.2.1

cd scipy-0.7.0
python setup.py config_fc --fcompiler=gfortran install --prefix=${HOME}/opt/python 2>&1 | tee install.log
cd ..
python -c "import scipy; scipy.test()"

Set these environment variables in the .bashrc file:

if [ -z "${PYTHONPATH}" ]
then
    export PYTHONPATH=""
fi

export PYTHONPATH=${HOME}/opt/python/lib64/python2.4/site-packages:${HOME}/opt/python/lib/python2.4/site-packages:${PYTHONPATH}

export OMPI=/usr/local/ompi-ifort
export OPAL_PREFIX=${OMPI}
export OMP_NUM_THREADS=1

if [ -z "${PATH}" ]
then
    export PATH=${OMPI}/bin
else
    export PATH=${OMPI}/bin:${PATH}
fi

if [ -z "${LD_LIBRARY_PATH}" ]
then
    export LD_LIBRARY_PATH=${OMPI}/lib
else
    export LD_LIBRARY_PATH=${OMPI}/lib:${LD_LIBRARY_PATH}
fi

Make sure that you have the right mpicc:

which mpicc
/usr/local/ompi-ifort/bin/mpicc

and build GPAW (python setup.py build_ext 2>&1 | tee build_ext.log) with this customize.py file (Note: instructions valid from the 5232 release):

scalapack = True

compiler = 'gcc'

extra_compile_args += [
    '-O3',
    '-funroll-all-loops',
    '-fPIC',
    ]

libraries= []

mkl_lib_path = '/opt/intel/mkl/10.0.4.023/lib/em64t/'

extra_link_args = [
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
mkl_lib_path+'libmkl_blacs_openmpi_lp64.a',
mkl_lib_path+'libmkl_scalapack.a',
mkl_lib_path+'libmkl_blacs_openmpi_lp64.a',
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
]

define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]

Note: is case of problems similar to those found on akka.hpc2n.umu.se static linking is required.

A gpaw script test.py can be submitted like this:

qsub -l nodes=1:ppn=8 -l walltime=00:30:00 -m abe run.sh

where run.sh looks like this:

#!/bin/sh

#PBS -m ae
#PBS -M email@email.com
#PBS -q long
#PBS -r n
#PBS -l nodes=1:ppn=8

cd $PBS_O_WORKDIR
echo Running on host `hostname` in directory `pwd`
NPROCS=`wc -l < $PBS_NODEFILE`
echo This jobs runs on the following $NPROCS processors:
cat $PBS_NODEFILE

export PYTHONPATH=${HOME}/opt/gpaw-0.7.5232:${HOME}/opt/python-ase-3.1.0.846:${PYTHONPATH}
export PYTHONPATH=${HOME}/opt/python/lib64/python2.4/site-packages:${HOME}/opt/python/lib/python2.4/site-packages:${PYTHONPATH}
export PATH=${HOME}/opt/gpaw-0.7.5232/build/bin.linux-x86_64-2.4:${PATH}
export GPAW_SETUP_PATH=${HOME}/opt/gpaw-setups-0.5.3574
export OMP_NUM_THREADS=1

mpiexec gpaw-python test.py

Please make sure that your jobs do not run multi-threaded, e.g. for a job running on node02 do from a login node:

ssh node02 ps -fL

you should see 1 in the NLWP column. Numbers higher then 1 mean multi-threaded job.

It’s convenient to customize as described on the Parallel runs page.