Differences between revisions 144 and 145
Revision 144 as of 2016-07-21 14:10:44
Size: 7673
Comment: Software modules on CentOS 7
Revision 145 as of 2018-04-06 14:19:24
Size: 147
Comment:
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
.. Contents::

This page documents the CAMd group's software packages installed on the Niflheim cluster.
To use this software you must log in to one of the login nodes, see `Getting_Started#login-to-niflheim`_.

Other user groups on Niflheim will have their own software installations, and this is not documented here.

Enabling modules
================

Software installed on Niflheim is managed using `modules <http://modules.sourceforge.net/>`_.

Modules are **not** activated by default. Users must make sure that modules are enabled:

- for **bash** or **ksh** shells the following files must be modified to include:

  - `~/.bashrc` (if using `bash`) or `~/.profile` (if using `ksh`)::

     if [ -r "/home/opt/modulefiles/modulefiles_el6.sh" ]; then
         source /home/opt/modulefiles/modulefiles_el6.sh
     fi
     export PYTHONDONTWRITEBYTECODE=1 # disable creation of pyc files

  - `~/.bash_profile` (if using `bash`) or `~/.profile` (if using `ksh`)::

     if [ -n "$PBS_ENVIRONMENT" ]; then
         cd $PBS_O_WORKDIR
     fi

- for **csh** or **tcsh** shells the following files must be modified to include:

  - `~/.cshrc` (if using `csh`) or `~/.tcshrc` (if using `tcsh`)::

     if ( -r "/home/opt/modulefiles/modulefiles_el6.csh" ) then
         source /home/opt/modulefiles/modulefiles_el6.csh
     endif
     setenv PYTHONDONTWRITEBYTECODE 1 # disable creation of pyc files

     if ($?PBS_ENVIRONMENT) then
         if ($PBS_ENVIRONMENT == PBS_BATCH) then
             cd $PBS_O_WORKDIR
         endif
     endif

    **Note** that **ALL** settings must be above the line (if present) in `~/.cshrc` or `~/.tcshrc`::

        if ( { tty -s } == 0 ) exit

    Moreover if both `~/.cshrc` or `~/.tcshrc` are present
    user **must** merge his personal settings from these two files into one (for example ~/.cshrc), and remove the other one.

The purpose of sourcing the `/home/opt/modulefiles/modulefiles.*` is to initialize the modules for the given type of shell,
and additionally set the **FYS_PLATFORM** environment variable which identifies the node and operating system type.
The command::

  grep -E "nodetype=|osversion=" /home/opt/modulefiles/modulefiles_el6.sh | sort | uniq

lists the nodetypes available on el6 (CentOS 6) operating system.
The **FYS_PLATFORM** environment variable is constructed as `nodetype-osversion`, print the current value of **FYS_PLATFORM** with::

  echo $FYS_PLATFORM

At his point one should be able to access the requested package by loading it using the **module load** mechanism, described below.

Using modules
=============

Modules provide a mechanism to set/unset all environment variables related to a given package in one UNIX shell command.

Type::

 module avail

to list all available modules on the **FYS_PLATFORM**.

Modules definitions are contained in so-called modulefiles located in several directories. The directory structure is dependent on the system.
Users shall focus on modules located under `/home/opt/modulefiles` and below, which are moreover easily recognizable
by their names entirely in **CAPITALS**. The recommended versions of packages are marked with **(default)**.

Verify which modules are loaded by::

 module list

In case when many modules need to be unloaded at once use::

 module purge

To make a package available, load the module::

 module load GPAW

The GPAW (note the **CAPITALS**) will load the gpaw (**lowercase**) module and all the dependencies.

To see the body of the module (i.e. the commands performed when the module is loaded) type::

 module show GPAW

and, when necessary, continue to the lower level (lowercase) modules::

 module show gpaw-setups

You can unload the module by::

 module unload GPAW

One can load also a specific version of software.
This is recommended due to the fact that the default version of software may change in the future::

 module load GPAW/0.9.0.8965-1

**Note**: it's sometimes useful to load a different version of the lower level (lowercase) modules.
List the available modules for ``gpaw-setups``::

 module avail gpaw-setups

and use unload/module load sequence, e.g.::

 module unload gpaw-setups
 module load gpaw-setups/0.9.9672

Permanent settings
------------------

To permanently enable a package one has to load the corresponding module in the shell login script.

Example below loads the default versions of `dacapo <https://wiki.fysik.dtu.dk/dacapo>`_ and `gpaw <https://wiki.fysik.dtu.dk/gpaw>`_.
One has to add the following to:

- `~/.bashrc` (if using `bash`)::

   if test -n "`echo $FYS_PLATFORM | grep el5`"; then
       module load DACAPO
       module load GPAW
   fi
   if test -n "`echo $FYS_PLATFORM | grep el6`"; then
       module load DACAPO
       module load GPAW
   fi

- `~/.cshrc` or `~/.tcshrc` (if using `tcsh`)::

   if ( "`echo $FYS_PLATFORM | grep el5`" != "" ) then
       module load DACAPO
       module load GPAW
   endif
   if ( "`echo $FYS_PLATFORM | grep el6`" != "" ) then
       module load DACAPO
       module load GPAW
   endif

**Note** that modules *prepend* environment variables: **PYTHONPATH**, **PATH**, **LD_LIBRARY_PATH**, **MANPATH**, etc.,
so any personal settings should be done after loading modules.

The GPAW module loads the version of ASE3 bound to the gpaw's release, so to get the latest ase
please load the ASE3 module after the GPAW module::

  module load GPAW
  module unload ASE3
  module load ASE3

Modules in batch jobs
---------------------

With permanent module settings described above submit a batch job with standard `qsub` command::

  qsub script.py

with an example `script.py`::

  #!/usr/bin/env python

  from ase import Hartree
  print Hartree

You can also enable the required modules for a given batch job
(depending on the shell this may work only when permanent modules settings related to those modules are **NOT** present in you shell startup scripts).
This feature is useful mostly for developers when comparing
different versions of a code, or for making sure that a given software will be used during the whole computational project.

Examples are given below:

- using jacapo interface with the latest **ASE3** with the following `jacapo.sh` script::

    #!/bin/sh

    if [ -r "/home/opt/modulefiles/modulefiles_el6.sh" ]; then
        source /home/opt/modulefiles/modulefiles_el6.sh
    fi
    if test -n "`echo $FYS_PLATFORM | grep el5`"; then
        module load DACAPO
        module load ASE3
        module unload SCIENTIFICPYTHON
        module load SCIENTIFICPYTHON/2.8
    fi
    if test -n "`echo $FYS_PLATFORM | grep el6`"; then
        module load DACAPO
        module unload ASE3
        module load ASE3
        module unload SCIENTIFICPYTHON
        module load SCIENTIFICPYTHON/2.8-`echo $FYS_PLATFORM | sed 's/-el6//'`-numpy-1.4.1-1
    fi

    python script.py

  You submit this script with::

    qsub jacapo.sh

Creating personal modules
-------------------------

The recommended place to keep personal modules is under **~/privatemodules** (on the Niflheim filesystem).
To enable this directory use the following command::

  module use --append ~/privatemodules

and to disable it::

  module unuse ~/privatemodules

Software modules on CentOS 7
============================

For the new (2017) CentOS 7 based Niflheim upgrade, we will convert to EasyBuild_modules_.
Please see the Niflheim7_Getting_started_ page.

Software installed on Niflheim

Please see the Niflheim7_Getting_started page.

Niflheim: Installed_software (last edited 2018-04-06 14:19:24 by OleHolmNielsen)