Software installed on Niflheim

This page documents the CAMd group's software packages installed on the Niflheim cluster. To use this software you must log in to one of the login nodes, see Getting_Started#login-to-niflheim.

Other user groups on Niflheim will have their own software installations, and this is not documented here.

Enabling modules

Software installed on Niflheim is managed using modules.

Modules are not activated by default. Users must make sure that modules are enabled:

  • for bash or ksh shells the following files must be modified to include:

    • ~/.bashrc (if using bash) or ~/.profile (if using ksh):

      if [ -r "/home/opt/modulefiles/modulefiles_el6.sh" ]; then
          source /home/opt/modulefiles/modulefiles_el6.sh
      fi
      export PYTHONDONTWRITEBYTECODE=1  # disable creation of pyc files
    • ~/.bash_profile (if using bash) or ~/.profile (if using ksh):

      if [ -n "$PBS_ENVIRONMENT" ]; then
          cd $PBS_O_WORKDIR
      fi
  • for csh or tcsh shells the following files must be modified to include:

    • ~/.cshrc (if using csh) or ~/.tcshrc (if using tcsh):

      if ( -r "/home/opt/modulefiles/modulefiles_el6.csh" ) then
          source /home/opt/modulefiles/modulefiles_el6.csh
      endif
      setenv PYTHONDONTWRITEBYTECODE 1  # disable creation of pyc files
      
      if ($?PBS_ENVIRONMENT) then
          if ($PBS_ENVIRONMENT == PBS_BATCH) then
              cd $PBS_O_WORKDIR
          endif
      endif

      Note that ALL settings must be above the line (if present) in ~/.cshrc or ~/.tcshrc:

      if ( { tty -s } == 0 ) exit

      Moreover if both ~/.cshrc or ~/.tcshrc are present user must merge his personal settings from these two files into one (for example ~/.cshrc), and remove the other one.

The purpose of sourcing the /home/opt/modulefiles/modulefiles.* is to initialize the modules for the given type of shell, and additionally set the FYS_PLATFORM environment variable which identifies the node and operating system type. The command:

grep -E "nodetype=|osversion=" /home/opt/modulefiles/modulefiles_el6.sh | sort | uniq

lists the nodetypes available on el6 (CentOS 6) operating system. The FYS_PLATFORM environment variable is constructed as nodetype-osversion, print the current value of FYS_PLATFORM with:

echo $FYS_PLATFORM

At his point one should be able to access the requested package by loading it using the module load mechanism, described below.

Using modules

Modules provide a mechanism to set/unset all environment variables related to a given package in one UNIX shell command.

Type:

module avail

to list all available modules on the FYS_PLATFORM.

Modules definitions are contained in so-called modulefiles located in several directories. The directory structure is dependent on the system. Users shall focus on modules located under /home/opt/modulefiles and below, which are moreover easily recognizable by their names entirely in CAPITALS. The recommended versions of packages are marked with (default).

Verify which modules are loaded by:

module list

In case when many modules need to be unloaded at once use:

module purge

To make a package available, load the module:

module load GPAW

The GPAW (note the CAPITALS) will load the gpaw (lowercase) module and all the dependencies.

To see the body of the module (i.e. the commands performed when the module is loaded) type:

module show GPAW

and, when necessary, continue to the lower level (lowercase) modules:

module show gpaw-setups

You can unload the module by:

module unload GPAW

One can load also a specific version of software. This is recommended due to the fact that the default version of software may change in the future:

module load GPAW/0.9.0.8965-1

Note: it's sometimes useful to load a different version of the lower level (lowercase) modules. List the available modules for gpaw-setups:

module avail gpaw-setups

and use unload/module load sequence, e.g.:

module unload gpaw-setups
module load gpaw-setups/0.9.9672

Permanent settings

To permanently enable a package one has to load the corresponding module in the shell login script.

Example below loads the default versions of dacapo and gpaw. One has to add the following to:

  • ~/.bashrc (if using bash):

    if test -n "`echo $FYS_PLATFORM | grep el5`"; then
        module load DACAPO
        module load GPAW
    fi
    if test -n "`echo $FYS_PLATFORM | grep el6`"; then
        module load DACAPO
        module load GPAW
    fi
  • ~/.cshrc or ~/.tcshrc (if using tcsh):

    if ( "`echo $FYS_PLATFORM | grep el5`" != "" ) then
        module load DACAPO
        module load GPAW
    endif
    if ( "`echo $FYS_PLATFORM | grep el6`" != "" ) then
        module load DACAPO
        module load GPAW
    endif

Note that modules prepend environment variables: PYTHONPATH, PATH, LD_LIBRARY_PATH, MANPATH, etc., so any personal settings should be done after loading modules.

The GPAW module loads the version of ASE3 bound to the gpaw's release, so to get the latest ase please load the ASE3 module after the GPAW module:

module load GPAW
module unload ASE3
module load ASE3

Modules in batch jobs

With permanent module settings described above submit a batch job with standard qsub command:

qsub script.py

with an example script.py:

#!/usr/bin/env python

from ase import Hartree
print Hartree

You can also enable the required modules for a given batch job (depending on the shell this may work only when permanent modules settings related to those modules are NOT present in you shell startup scripts). This feature is useful mostly for developers when comparing different versions of a code, or for making sure that a given software will be used during the whole computational project.

Examples are given below:

  • using jacapo interface with the latest ASE3 with the following jacapo.sh script:

    #!/bin/sh
    
    if [ -r "/home/opt/modulefiles/modulefiles_el6.sh" ]; then
        source /home/opt/modulefiles/modulefiles_el6.sh
    fi
    if test -n "`echo $FYS_PLATFORM | grep el5`"; then
        module load DACAPO
        module load ASE3
        module unload SCIENTIFICPYTHON
        module load SCIENTIFICPYTHON/2.8
    fi
    if test -n "`echo $FYS_PLATFORM | grep el6`"; then
        module load DACAPO
        module unload ASE3
        module load ASE3
        module unload SCIENTIFICPYTHON
        module load SCIENTIFICPYTHON/2.8-`echo $FYS_PLATFORM | sed 's/-el6//'`-numpy-1.4.1-1
    fi
    
    python script.py

    You submit this script with:

    qsub jacapo.sh

Creating personal modules

The recommended place to keep personal modules is under ~/privatemodules (on the Niflheim filesystem). To enable this directory use the following command:

module use --append ~/privatemodules

and to disable it:

module unuse ~/privatemodules

Software modules on CentOS 7

For the new (2017) CentOS 7 based Niflheim upgrade, we will convert to EasyBuild_modules.