Cluster software

Cloning of nodes

The NIFLHEIM cluster uses the SystemImager toolkit on a central server to create an image of a Golden Client node that has been installed in the usual way using a distribution on CD-ROM (Centos Linux in our case). The SystemImager is subsequently used to install identical images of the Golden Client on all of the nodes (changing of course hostname and network parameters).

Pathscale compilers

On the frontend computer slid the Pathscale compiler suite is installed. This includes:

  • C (pathcc) and C++ (pathCC)
  • Fortran (pathf90 and pathf77)

Libraries and include files for the compilers are installed on the compute nodes in:

/opt/pathscale

NetCDF (network Common data Form)

NetCDF (network Common Data Form) is an interface for array-oriented data access and a library that provides an implementation of the interface. The netCDF library also defines a machine-independent format for representing scientific data.

NetCDF is installed on the cluster via the rpm netcdf-3.6.1-1.2.el4.fys. This build includes bindings to the PathScale fortran compiler. Spec file for the rpm is in:

~rpmbild/SPECS/netcdf.spec

OpenMPI (A High Performance Message Passing Library)

The version now installed is 1.1.1.

Get the buildrpms.sh script from the page.

Build OpenMPI with support for:

  • Torque (installation in /usr/local)
  • gcc c compiler
  • pathscale fortran compiler

This is done by modifying the buildrpms.sh script. Change the following lines:

prefix="/usr/local"
configure_options="--with-tm=/usr/local FC=pathf90 F77=pathf90"

buildrpms.sh is used to build one single rpm, including man pages:

build_srpm=no
build_single=yes
build_multiple=no

The build of multiple rpms did not seem to work in version 1.1.1.

buildrpms.sh needs the rpm spec file, this can be copied from the unpacked openmpi tar file:

cp openmpi-1.1.1/contrib/dist/linux/openmpi.spec .

Now run:

./buildrpms.sh openmpi-<version>.tar.gz

Copy the generated rpm in:

/home/niflheim/lhansen/RPMS/openmpi-<version>.rpm

to the appropriate yum repository:

REPO/redhat/linux/4/x86_64/fys/

and update the yum repository home/camp/rpmbuild/update-repo.

Now run as root yum update.

Check the installation using:

ompi_info | grep tm
      MCA memory: ptmalloc2 (MCA v1.0, API v1.0, Component v1.1)
      MCA ras: tm (MCA v1.0, API v1.0, Component v1.1)
      MCA pls: tm (MCA v1.0, API v1.0, Component v1.1)

Processor affinity

For some parallel libraries like BLACS and ScaLAPACK which are using OpenMPI as the communication layer, setting processor affinity seems to be important. Processor affinity is when a process is fixed on a specific processor, see more details in this OpenMPI FAQ.

To enable processor affinity use:

mpirun --mca mpi_paffinity_alone 1 -np 4 a.out

Without this option all copies of the parallel program (siesta) would land on one CPU.