cori.nersc.gov (XC40)

Note

These instructions are up-to-date as of April 2021.

GPAW

At NERSC it is recommened to install GPAW on Cori with Anaconda python. For massivly parallel applications it is recommened to use Shifter.

GPAW can be built with a minimal siteconfig.py

parallel_python_interpreter = True
compiler = 'cc'
mpicompiler = 'cc'
mpilinker = 'cc'
scalapack = True
libxc = '/usr/common/software/libxc/4.2.3/gnu/haswell'
include_dirs += [libxc + '/include']
library_dirs += [libxc + '/lib']
extra_link_args += [f'-Wl,-rpath={libxc}/lib']
extra_compile_args += ['-O2']
if 'xc' not in libraries:
    libraries.append('xc')
# these are in the cray wrapper
if 'blas' in libraries:
    libraries.remove('blas')
if 'lapack' in libraries:
    libraries.remove('lapack')

if scalapack:
    define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
    define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]

Load the GNU programming environment and set Cray environment for dynamic linking:

export CRAYPE_LINK_TYPE=dynamic
module swap PrgEnv-${PE_ENV,,} PrgEnv-gnu
module load python

Create a conda environment for gpaw:

conda create --name gpaw python=3.8 pip numpy scipy matplotlib

Install ASE with pip while the Anaconda python module is loaded:

source activate gpaw
pip install ase

Build and install GPAW:

python setup.py build_ext
python setup.py install

To setup the environment:

module swap PrgEnv-${PE_ENV,,} PrgEnv-gnu
module load python
source activate gpaw
export OMP_NUM_THREADS=1

Then the test suite can be run from a batch script or interactive session with:

export MKL_CBWR="AVX"
srun -n 8 -c 2 --cpu_bind=cores gpaw-python -m gpaw test

Note

For all tests to pass enable MKL’s conditional Numerical Reproducibility mode with the \(MKL_CBWR\) environment variable.

Note

To run the test suite you will first need the GPAW setups installed, as described at Installation of PAW datasets.