The lumi.csc.fi supercomputer

Note

These instructions are up-to-date as of January 2023.

As all the GPU related libraries are not available in the CPU only partition, separate installation is needed for LUMI-C and LUMI-G.

GPAW for LUMI-C

Following instructions are preliminary guidelines on how to install GPAW to be utilized with the CPU only partition of LUMI-C. To begin, add a following file to your current directory and call it siteconfig-lumi.py.

# Custom GPAW siteconfig for LUMI

# compiler and linker 
compiler = 'cc'
mpicompiler = 'cc'
mpilinker = 'cc'
extra_compile_args = ['-g', '-fPIC', '-march=native', '-O3', '-fopenmp']
extra_link_args = ['-fopenmp']

# ScaLAPACK 
# required libraries are linked by compiler wrappers 
scalapack = True

# libxc 
xc = '/appl/lumi/spack/22.08/0.18.1/opt/spack/libxc-5.1.7-c55ppsw/'
include_dirs += [xc + 'include']
library_dirs += [xc + 'lib']
# Use rpath to avoid setting LD_LIBRARY_PATH:
extra_link_args += ['-Wl,-rpath={xc}/lib'.format(xc=xc)]
if 'xc' not in libraries:
    libraries.append('xc')

Following script will install latest GPAW under gpaw-cpu sub-directory (within the current directory). It is recommended to perform installation under the project directory (see LUMI user documentation):

module load cray-python/3.9.12.1 PrgEnv-gnu/8.3.3

export GPAW_CONFIG=$PWD/siteconfig-lumi.py
export PYTHONUSERBASE=$PWD/gpaw-cpu
pip install --user git+https://gitlab.com/gpaw/gpaw.git 2>&1 | tee install.log

In order to use GPAW, one needs to set also PATH:

export PATH=$PYTHONUSERBASE/bin:$PATH

GPAW for LUMI-G

GPU version of GPAW depends on cupy which needs to be installed first. Currently, the version of rocm library differs in the login and compute nodes, and thus the installation needs to be done in the compute node as follows:

module load cray-python/3.9.12.1 PrgEnv-gnu/8.3.3
module load craype-accel-amd-gfx90a rocm

export PYTHONUSERBASE=$PWD/gpaw-gpu
# Start shell in GPU node
srun -p dev-g --nodes=1 --ntasks-per-node=1 --gpus-per-node=1 -A <project> -t 0:30:00 --pty bash
# Set environment variables for cupy installation
export CUPY_INSTALL_USE_HIP=1
export ROCM_HOME=$ROCM_PATH
export HCC_AMDGPU_TARGET=gfx90a
pip install --user git+https://github.com/cupy/cupy.git@v11.2.0

Configuring MyQueue

Use the following MyQueue config.py file:

config = {
    'scheduler': 'slurm',
    'extra_args': ['--account=project_462000135', '--mem=0'],
    'mpiexec': 'srun',
    'parallel_python': 'gpaw python',
    'nodes': [
        ('standard', {'cores': 128}),
        ('small', {'cores': 128}),
        ('debug', {'cores': 128}),
        ('largemem', {'cores': 128}),
        ('standard-g', {'cores': 63}),
        ('small-g', {'cores': 63}),
        ('dev-g', {'cores': 63})]}

and submit jobs like this:

mq submit job.py -R 128:standard:2h