The lumi.csc.fi supercomputer

Note

These instructions are up-to-date as of September 2023.

It is recommended to perform installation under the /projappl/project_... directory (see LUMI user documentation). A separate installation is needed for LUMI-C and LUMI-G.

GPAW for LUMI-G

Load the following modules:

export EBU_USER_PREFIX=/scratch/project_465000538/GPAW/EasyBuild
module load LUMI/22.12 partition/G
module load cpeGNU/22.12
module load craype-accel-amd-gfx90a
module load rocm/5.2.3
module load cray-python/3.9.13.1
module load cray-fftw/3.3.10.1
module load ASE/3.22.1-cpeGNU-22.12
module load CuPy/12.2.0-cpeGNU-22.12
module load ELPA/2023.05.001-cpeGNU-22.12-GPU
module load libxc/6.2.2-cpeGNU-22.12

Create a virtual environment and activate it:

python3 -m venv venv
source venv/bin/activate

Clone the GPAW source code:

git clone git@gitlab.com:gpaw/gpaw

Copy this siteconfig-lumi-gpu.py to gpaw/siteconfig.py and compile the C-code and the GPU kernels with:

pip install -v -e gpaw/

Now insert the export EBU_USER_PREFIX=... line and all the module load lines from above into the start of your venv/bin/activate script so that the modules are always loaded when you activate your new environment.

Interactive jobs can be run like this:

srun -A project_465000538 -p small-g --nodes=1 --ntasks-per-node=2 --gpus-per-node=1 -t 0:30:00 --pty bash

To use Omnitrace, source this file???:

source /scratch/project_465000538/GPAW/omnitrace-1.10.2-opensuse-15.4-ROCm-50200-PAPI-OMPT-Python3/share/omnitrace/setup-env.sh

Configuring MyQueue

Use the following MyQueue config.py file:

config = {
    'scheduler': 'slurm',
    'extra_args': ['--account=project_4xxxxxxxx', '--mem=0'],
    'mpiexec': 'srun',
    'parallel_python': 'gpaw python',
    'nodes': [
        ('standard', {'cores': 128}),
        ('small', {'cores': 128}),
        ('debug', {'cores': 128}),
        ('largemem', {'cores': 128}),
        ('standard-g', {'cores': 63}),
        ('small-g', {'cores': 63}),
        ('dev-g', {'cores': 63})]}

and submit jobs like this:

mq submit job.py -R 128:standard:2h