Curie (BullX cluster, Intel Nehalem, Infiniband QDR, MKL)

Note

These instructions are up-to-date as of October 2014

Here you find information about the system http://www-hpc.cea.fr/en/complexe/tgcc-curie.htm

For large calculations, it is suggested that one utilizes the Scalable Python interpreter. Small to medium size calculations are fine with standard Python, for these one can use system’s default Python (which contains NumPy), thus one can skip directly to LibXC/GPAW instructions.

Scalable Python

Standard Python interpreter has serious bottleneck in large scale parallel calculations when many MPI tasks perform the same I/O operations during the import statetements. Scalable Python https://gitorious.org/scalable-python reduces the import time by having only single MPI task to perform import related I/O operations and using then MPI for broadcasting the data.

First, download the source code and switch to GNU environment:

git clone git@gitorious.org:scalable-python/scalable-python.git
module switch intel gnu
export OMPI_MPICC=gcc

Use the following build script (change the installation prefix to a proper one):

#!/bin/bash

export CC=gcc
export CXX=g++
export MPICC=mpicc
# export XTPE_LINK_TYPE=dynamic
export LINKFORSHARED='-Wl,-export-dynamic -dynamic'
export MPI_LINKFORSHARED='-Wl,-export-dynamic -dynamic'

install_prefix=/my_prefix/scalable-gnu

# Make zlib built-in to the interpreter
sed -i -e 's/^#zlib.*/zlib zlibmodule.c -lz/' Modules/Setup.dist

./configure --prefix=$install_prefix --enable-mpi --disable-ipv6 2>&1 | tee log-conf

make 2>&1 | tee log-make
make install 2>&1 | tee log-inst

make mpi 2>&1 | tee log-make-mpi
make install-mpi 2>&1 | tee log-inst-mpi

Add then install_prefix/bin to your PATH, and download and install NumPy:

export PATH=install_prefix/bin:$PATH
wget http://sourceforge.net/projects/numpy/files/NumPy/1.8.1/numpy-1.8.1.tar.gz
tar xzf numpy-1.8.1.tar.gz
cd numpy-1.8.1
python setup.py install

LibXC

Download libxc:

wget http://www.tddft.org/programs/octopus/down.php?file=libxc/libxc-2.2.0.tar.gz

Configure and make:

./configure --prefix=install_prefix CFLAGS=-fPIC
make
make install

GPAW

Intel compiler gives a bit better performance for GPAW, so one should switch back to Intel environment after Scalable Python/LibXC installation:

module switch gnu intel
unset OMPI_MPICC

Furthermore, in order to utilize HDF5 load the module:

module load hdf5/1.8.12_parallel

Use the compiler wrapper file icc.py

#!/usr/bin/python
"""icc.py is a wrapper for the Intel compiler,
   converting/removing incompatible gcc args.   """

import sys
from subprocess import call

args2change = {"-fno-strict-aliasing":"",
               "-fmessage-length=0":"",
               "-Wall":"",
               "-std=c99":"-qlanglvl=extc99",
               "-fPIC":"",
               "-g":"",
               "-D_FORTIFY_SOURCE=2":"",
               "-DNDEBUG":"",
               "-UNDEBUG":"",
               "-pthread":"",
               "-shared":"-qmkshrobj",
               "-Xlinker":"",
               "-export-dynamic":"",
               "-Wstrict-prototypes":"",
               "-dynamic":"",
               "-O3":"",
               "-O3":"",
               "-O2":"",
               "-O1":""}

fragile_files = ["test.c"]

cmd = ""
fragile = False
for arg in sys.argv[1:]:
    cmd += " "
    t = arg.strip()
    if t in fragile_files:
        fragile = True
    if t in args2change:
        cmd += args2change[t]
    else:
        cmd += arg

flags = "-w -O3 -std=c99"
cmd = f"mpicc {flags} {cmd}"

call(cmd, shell=True)

and the following configuration file customize_curie_icc.py.

import os

extra_compile_args = ['-std=c99']
compiler = './icc.py'
mpicompiler = './icc.py'
mpilinker = './icc.py'

#libxc
library_dirs = ['/ccc/cont005/home/pawp72/enkovaaj/libxc/lib']
include_dirs += ['/ccc/cont005/home/pawp72/enkovaaj/libxc/include']
libraries = ['z', 'xc']

scalapack = True
hdf5 = True

mkl_flags = os.environ['MKL_SCA_LIBS']
extra_link_args = [mkl_flags]

# HDF5 flags
include_dirs += [os.environ['HDF5_INCDIR']]
libraries += ['hdf5']
library_dirs += [os.environ['HDF5_LIBDIR']]

define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]

GPAW can now build in a normal way (make sure Scalable Python is in the PATH):

python setup.py install --home=path_to_install_prefix