Curie (BullX cluster, Intel Nehalem, Infiniband QDR, MKL)


These instructions are up-to-date as of October 2014

Here you find information about the the system

For large calculations, it is suggested that one utilizes the Scalable Python interpreter. Small to medium size calculations are fine with standard Python, for these one can use system’s default Python (which contains NumPy), thus one can skip directly to LibXC/GPAW instructions.

Scalable Python

Standard Python interpreter has serious bottleneck in large scale parallel calculations when many MPI tasks perform the same I/O operations during the import statetements. Scalable Python reduces the import time by having only single MPI task to perform import related I/O operations and using then MPI for broadcasting the data.

First, download the source code and switch to GNU environment:

git clone
module switch intel gnu
export OMPI_MPICC=gcc

Use the following build script (change the installation prefix to a proper one):


export CC=gcc
export CXX=g++
export MPICC=mpicc
# export XTPE_LINK_TYPE=dynamic
export LINKFORSHARED='-Wl,-export-dynamic -dynamic'
export MPI_LINKFORSHARED='-Wl,-export-dynamic -dynamic'


# Make zlib built-in to the interpreter
sed -i -e 's/^#zlib.*/zlib zlibmodule.c -lz/' Modules/Setup.dist

./configure --prefix=$install_prefix --enable-mpi --disable-ipv6 2>&1 | tee log-conf

make 2>&1 | tee log-make
make install 2>&1 | tee log-inst

make mpi 2>&1 | tee log-make-mpi
make install-mpi 2>&1 | tee log-inst-mpi

Add then install_prefix/bin to your PATH, and download and install NumPy:

export PATH=install_prefix/bin:$PATH
tar xzf numpy-1.8.1.tar.gz
cd numpy-1.8.1
python install


Download libxc:


Configure and make:

./configure --prefix=install_prefix CFLAGS=-fPIC
make install


Intel compiler gives a bit better performance for GPAW, so one should switch back to Intel environment after Scalable Python/LibXC installation:

module switch gnu intel

Furthermore, in order to utilize HDF5 load the module:

module load hdf5/1.8.12_parallel

Use the compiler wrapper file

""" is a wrapper for the Intel compiler,
   converting/removing incompatible gcc args.   """

import sys
from subprocess import call

args2change = {"-fno-strict-aliasing":"",

fragile_files = ["test.c"]

cmd = ""
fragile = False
for arg in sys.argv[1:]:
    cmd += " "
    t = arg.strip()
    if t in fragile_files:
        fragile = True
    if t in args2change:
        cmd += args2change[t]
        cmd += arg

flags = "-w -O3 -std=c99"
cmd = "mpicc %s %s"%(flags, cmd)

call(cmd, shell=True)

and the following configuration file

import os

extra_compile_args = ['-std=c99']
compiler = './'
mpicompiler = './'
mpilinker = './'

library_dirs = ['/ccc/cont005/home/pawp72/enkovaaj/libxc/lib']
include_dirs += ['/ccc/cont005/home/pawp72/enkovaaj/libxc/include']
libraries = ['z', 'xc']

scalapack = True
hdf5 = True

mkl_flags = os.environ['MKL_SCA_LIBS']
extra_link_args = [mkl_flags]

# HDF5 flags
include_dirs += [os.environ['HDF5_INCDIR']]
libraries += ['hdf5']
library_dirs += [os.environ['HDF5_LIBDIR']]

define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]

GPAW can now build in a normal way (make sure Scalable Python is in the PATH):

python install --home=path_to_install_prefix