jaguar (Cray XT5)

Here you find information about the system

The current operating system in Cray XT4/XT5 compute nodes, Compute Linux Environment (CLE) has some limitations, most notably it does not support shared libraries. In order to use python in CLE some modifications to the standard python are needed. Instructions below assume tcsh.

The installations process of python and numpy can be performed with the script make_python_numpy:

./make_python_numpy |& tee all.log

whose details are given below. Note: One may want to change the installation paths in the beginning of make_python_numpy.

Set the correct C compiler and flags, e.g.:

module swap PrgEnv-pgi PrgEnv-gnu
setenv CC cc
setenv CXX CC
setenv OPT '-O3 -funroll-all-loops'

The following modules are loaded:

module avail
 Currently Loaded Modulefiles:
 1) modules/3.1.6
 2) DefApps
 3) torque/2.2.0-snap.200707311754
 4) moab/5.2.4
 5) xtpe-quadcore
 6) MySQL/5.0.45
 7) xt-service/2.1.50HD
 8) xt-libc/2.1.50HD
 9) xt-os/2.1.50HD
10) xt-boot/2.1.50HD
11) xt-lustre-ss/2.1.50HD.PS04.lus.1.6.5.steve.8103_1.6.5
12) xtpe-target-cnl
13) Base-opts/2.1.50HD
14) PrgEnv-gnu/2.1.50HD
15) xt-asyncpe/2.3
16) xt-pe/2.1.50HD
17) xt-mpt/3.1.0
18) xt-libsci/10.3.1
19) fftw/3.1.1
20) gcc/4.2.0.quadcore

The recommended place for user’s applications is under $HOME:

mkdir -p sw/xt5
cd sw/xt5
set sw_home=~/sw/xt5
tar jxf Python-2.5.4.tar.bz2
tar zxf expat-2.0.1.tar.gz
tar jxf zlib-1.2.3.tar.bz2
tar zxf nose-0.11.0.tar.gz
tar zxf numpy-1.2.1.tar.gz

Before installing a special python, expat and zlib which are needed by GPAW, but which are not included in the python distribution. The installation is based on instructions from

Install expat:

cd ${sw_home}
setenv EXPAT_DIR ${sw_home}/expat-2.0.1-1
cd expat-2.0.1
./configure --disable-shared --prefix=${EXPAT_DIR}
make install

Install zlib:

cd ${sw_home}
setenv ZLIB_DIR ${sw_home}/zlib-1.2.3-1
cd zlib-1.2.3
./configure --prefix=${ZLIB_DIR}
make # ignore error: /usr/lib/../lib64/libc.a: could not read symbols: Bad value
make install

Next, one can proceed with the actual python installation. The following instructions are tested with python 2.5.4:

  • enter the python source directory:

    cd ${sw_home}
    setenv PYTHON_DIR ${sw_home}/Python-2.5.4-1
    cd Python-2.5.4
  • create a special dynamic loader for correct resolution of namespaces:

    wget --no-check-certificate -O Python/dynload_jaguar.c
  • run configure:

    ./configure --prefix=${PYTHON_DIR} SO=.a DYNLOADFILE=dynload_jaguar.o MACHDEP=jaguar --host=x86_64-unknown-linux-gnu --disable-sockets --disable-ssl --enable-static --disable-shared | tee config.log
  • in order to use distutils append the Lib/distutils/ file, so that static libraries are created instead of shared ones:

    wget --no-check-certificate
    cat Lib/distutils/ >
    mv  Lib/distutils
  • specify which modules will be statically linked in to the python interpreter by editing Modules/Setup:

    mv Modules/Setup Modules/Setup.orig
    wget --no-check-certificate -O Modules/Setup
    touch Modules/Setup

    Note: sha modules are required by numpy, so the following lines should be present in Modules/Setup:

    _sha shamodule.c
    _sha256 sha256module.c
    _sha512 sha512module.c
  • modify Lib/ as described at (is it really needed?),

  • build and install:

    make | tee make.log
    # ignore errors like:
    # *** WARNING: renaming "_ctypes" since importing it failed: dynamic module does not define init function (init_ctypes)
    make install | tee make_install.log
  • build numpy:

    cd ${sw_home}
    cd numpy-1.2.1
    ${PYTHON_DIR}/bin/python install | tee install.log

    Note: numpy 1.3.0 gives:

    # ImportError: No module named select
  • append numpy to pythons’s Modules/Setup:

    cd ${sw_home}/Python-2.5.4
    cat ../numpy-1.2.1/install.log | grep Append | cut -d ":" -f 2 | sed -n 's/ *//p' > append
    cat Modules/Setup append > Setup
    mv Setup Modules

    example output:

    cat append
    multiarray /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/core/multiarray.a
    umath /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/core/umath.a
    _sort /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/core/_sort.a
    scalarmath /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/core/scalarmath.a
    _compiled_base /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/lib/_compiled_base.a
    _capi /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/numarray/_capi.a
    fftpack_lite /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/fft/fftpack_lite.a
    lapack_lite /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.a
    mtrand /autofs/na1_home/farberow/sw/xt5/numpy-1.2.1/build/lib.linux-x86_64-2.5/numpy/random/mtrand.a
  • rebuild python:

    make | tee make2.log
    make install | tee make_install2.log

On jaguar only /tmp/work/$USER filesystem is available for batch jobs. Note: that this space is cleaning periodically Test python/numpy:

cp -r ${PYTHON_DIR} /tmp/work/$USER
cp -r ${sw_home}/nose-0.11.0 /tmp/work/$USER
cd /tmp/work/$USER

cat <<EOF > ./
import numpy
from numpy.core.multiarray import dot
b = numpy.ones(13, numpy.complex)
d = dot(b, b)

cat <<EOF > ./numpyTest.pbs
#PBS -l walltime=00:10:00,size=8
#PBS -N numpyTest
#PBS -j oe

export PYTHONHOME=/tmp/work/$USER/Python-2.5.4-1
export PYTHONPATH=/tmp/work/$USER/nose-0.11.0

cd /tmp/work/$USER
env | grep PYTHON
env | grep LD_LIBRARY_PATH
aprun -n1  ${PYTHONHOME}/bin/python -v ./

qsub numpyTest.pbs

Install ase/gpaw-setups (Note: use the latest releases):

cd ${sw_home}
wget --no-check-certificate
tar zxf python-ase-
wget --no-check-certificate
tar zxf gpaw-setups-0.5.3574.tar.gz

cp -r python-ase- gpaw-setups-0.5.3574 /tmp/work/$USER
cd /tmp/work/$USER
ln -s python-ase- ase

Install gpaw (Note: instructions valid from the 5232 release):

cd ${sw_home}
wget --no-check-certificate
tar zxf gpaw-0.7.5232.tar.gz
cd gpaw-0.7.5232
wget --no-check-certificate -O
${PYTHON_DIR}/bin/python build_ext | tee build_ext.log
cp -r ${sw_home}/gpaw-0.7.5232 /tmp/work/$USER
cd /tmp/work/$USER
ln -s gpaw-0.7.5232 gpaw

Test gpaw:

cd /tmp/work/$USER

cat <<EOF > ./gpawTest.pbs
#PBS -l walltime=00:40:00,size=8
#PBS -N gpawTest
#PBS -j oe

export PYTHONHOME=/tmp/work/$USER/Python-2.5.4-1
export GPAW_SETUP_PATH=/tmp/work/$USER/gpaw-setups-0.5.3574
export PYTHONPATH=/tmp/work/$USER/gpaw:/tmp/work/$USER/ase

cd /tmp/work/$USER/gpaw/gpaw/test
env | grep PYTHON
env | grep LD_LIBRARY_PATH
aprun -n4 /tmp/work/$USER/gpaw/build/bin.linux-x86_64-2.5/gpaw-python -v ./

qsub gpawTest.pbs