cori.nersc.gov (XC40)

Note

These instructions are up-to-date as of July 2018.

GPAW

At NERSC it is recommened to install GPAW on Cori with Anaconda python. For massivly parallel applications it is recommened to use Shifter.

GPAW can be built with a minimal customize.py

compiler = 'cc'
mpicompiler = 'cc'
mpilinker = 'cc'
scalapack=True
libxc='/usr/common/software/libxc/4.2.3/gnu/haswell'
include_dirs += [libxc+'/include']
library_dirs += [xc + '/lib']
extra_link_args += ['-Wl,-rpath={xc}/lib'.format(xc=libxc)]
extra_compile_args += ['-O2']
if 'xc' not in libraries:
    libraries.append('xc')
# these are in the cray wrapper
if 'blas' in libraries:
    libraries.remove('blas')
if 'lapack' in libraries:
    libraries.remove('lapack')

if scalapack:
    define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
    define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]

Load the GNU programming environment and set Cray environment for dynamic linking:

export CRAYPE_LINK_TYPE=dynamic
module swap ${PE_ENV,,} PrgEnv-gnu
module load python

Create a conda environment for gpaw:

conda create --name gpaw python=3.6 pip numpy scipy matplotlib

Install ASE with pip while the Anaconda python module is loaded:

source activate gpaw
pip install ase

Build and install GPAW:

python setup.py build_ext
python setup.py install

To setup the environment:

module swap ${PE_ENV,,} PrgEnv-gnu
module load python
source activate gpaw
export OMP_NUM_THREADS=1

Then the test suite can be run from a batch script or interactive session with:

export MKL_CBWR="AVX"
srun -n 8 -c 2 --cpu_bind=cores gpaw-python -m gpaw test

Note

For all tests to pass enable MKL’s conditional Numerical Reproducibility mode with the \(MKL_CBWR\) environment variable.