Sophia cluster at HPC Risø¶

This document explains how to compile a developer version of GPAW on Sophia.

Creating the venv¶

Download the sophia-venv.sh script and run it like this:

$ ./sophia-venv.sh <venv-name>
...

After a few minutes, you will have a <venv-name> folder with a GPAW installation inside.

Using the venv¶

In the following, we will assume that your venv folder is ~/venv1/. The venv needs to be activated like this:

$ source ~/venv1/bin/activate

and you can deactivate it when you no longer need to use it:

$ deactivate

You will want the activation to happen automatically for the jobs you submit to Sophia. Here are three ways to do it:

If you always want to use one venv then just put the activation command in your ~/.bashrc.
If you only want jobs running inside a certain folder to use the venv, then add this to your ~/.bashrc:
```
if [[ $SLURM_SUBMIT_DIR/ = $HOME/project-1* ]]; then
    source ~/venv1/bin/activate
fi
```
Now, SLURM-jobs submitted inside your ~/project-1/ folder will use the venv.
Use the “automatic discovery of venv’s” feature of MyQueue:
```
$ cd ~/project-1
$ ln -s ~/venv1 venv
$ mq submit job.py
```
MyQueue will look for venv/ folders (or soft-links as in the example) in one of the parent folders and activate the venv automatically when your job starts running.

Full script¶

#!/usr/bin/bash
set -e  # stop if there are errors
NAME=$1
FOLDER=$PWD

echo '
export EASYBUILD_PREFIX=/groups/physics/modules
module use $EASYBUILD_PREFIX/modules/all
module purge
unset PYTHONPATH
module load GPAW/21.6.0-foss-2020b-libxc-5.1.5-ASE-3.22.0
' > modules.sh
. modules.sh

python3 -m venv $NAME
cd $NAME
VENV=$PWD
. bin/activate
PIP="python3 -m pip"
$PIP install --upgrade pip -qq

mv bin/activate old
mv ../modules.sh bin/activate
cat old >> bin/activate
rm old
$PIP install pytz
git clone https://gitlab.com/ase/ase.git
$PIP install -U ase/

git clone https://gitlab.com/asr-dev/asr.git
cd asr
git checkout old-master
#git checkout om_spinspiral
cd ..
$PIP install -e asr

git clone https://gitlab.com/gpaw/gpaw.git
cd gpaw
#git checkout old-spinspiral
cd ..
echo "
from os import environ
from pathlib import Path
scalapack = True
fftw = True
libraries = ['openblas', 'fftw3', 'readline', 'gfortran',
             'scalapack', 'xc', 'vdwxc']
libxc = Path(environ['EBROOTLIBXC'])
include_dirs.append(libxc / 'include')
libvdwxc = Path(environ['EBROOTLIBVDWXC'])
include_dirs.append(libvdwxc / 'include')
library_dirs = environ['LD_LIBRARY_PATH'].split(':')
" > gpaw/siteconfig.py
pip install -e gpaw

$PIP install myqueue graphviz qeh ase-ext

gpaw install-data --basis --version=20000 . --no-register
export GPAW_SETUP_PATH=$GPAW_SETUP_PATH:$VENV/gpaw-basis-pvalence-0.9.20000
echo "export GPAW_SETUP_PATH=$GPAW_SETUP_PATH" >> bin/activate

ase completion >> bin/activate
gpaw completion >> bin/activate
mq completion >> bin/activate
$PIP completion --bash >> bin/activate

echo '
if [[ $SLURM_SUBMIT_DIR ]]; then
    export MPLBACKEND=Agg
else
    export MPLBACKEND=TkAgg
fi' >> bin/activate

pip install flask==2.1.0

# Fix to remove the GPAW and ASE module installed in the "GPAW/21.6.0-foss-2020b" toolchain from path
echo '
export PYTHONPATH=/groups/physics/modules/software/spglib-python/1.16.0-foss-2020b/lib/python3.8/site-packages:/groups/physics/modules/software/matplotlib/3.3.3-foss-2020b/lib/python3.8/site-packages:/g\
roups/physics/modules/software/Pillow/8.0.1-GCCcore-10.2.0/lib/python3.8/site-packages:/groups/physics/modules/software/Tkinter/3.8.6-GCCcore-10.2.0/lib/python3.8:/groups/physics/modules/software/Tkinte\
r/3.8.6-GCCcore-10.2.0/easybuild/python:/groups/physics/modules/software/SciPy-bundle/2020.11-foss-2020b/lib/python3.8/site-packages:/groups/physics/modules/software/pybind11/2.6.0-GCCcore-10.2.0/lib/py\
thon3.8/site-packages:/groups/physics/modules/software/Python/3.8.6-GCCcore-10.2.0/easybuild/python
' >> bin/activate

mq --version
#ase info
#gpaw test
source bin/activate
pip uninstall ase
pip install ase/
ase info
gpaw test