Differences between revisions 88 and 89
Revision 88 as of 2008-07-11 12:17:55
Size: 21696
Editor: MarcinDulak
Comment:
Revision 89 as of 2008-07-11 12:27:18
Size: 21973
Editor: MarcinDulak
Comment:
Deletions are marked like this. Additions are marked like this.
Line 117: Line 117:
Get the pseudopotentials tar-ball here: campos-dacapo-pseudopotentials-1.tar.gz_.

.. _campos-dacapo-pseudopotentials-1.tar.gz: attachment:campos-dacapo-pseudopotentials-1.tar.gz
Line 121: Line 125:
    cp psp/*/*/*.pseudo /some/directory/     cp *.pseudo /some/directory/
Line 148: Line 152:
   a) Unpack the tar file_ and::

        cd Dacapo

   b
) svn (the commands below will fetch the whole dacapo fortran and python source, and pseudopotentials)::
   a) svn (the commands below will fetch the whole dacapo fortran and python source, and pseudopotentials)::
Line 159: Line 159:


   b) alternatively, get the tar-ball (usually slightly outdated) here: Dacapo-0.9.3.tar.gz_.

.. _Dacapo-0.9.3.tar.gz: attachment:Dacapo-0.9.3.tar.gz

1   Download and Installation

1.2   Installation

Preferably build from our SRPMS (CentOS 4/5 and Fedora Core 7/8/9): see Cluster software RPMS.

If you prefer to install from tarballs the next section describes how the fortran program, pseudopotentials and python interface can be installed.

If you have an installation using the old CamposASE, it is enough just installing the new python interface, see Dacapo Python interface.

1.2.1   Dacapo fortran program and pseudopotentials

This section describes how the fortran program and pseudopotential are installed. The current version of the fortran program is 2.7.14.

1.2.2   From svn

This is the recommended method. The commands below will fetch the whole dacapo fortran and python source, and pseudopotentials:

cd
svn checkout https://USER@svn.fysik.dtu.dk/projects/dacapo/trunk dacapo

where "USER@" part has to be skipped for anonymous access.

Alternatively, get the tar-ball (usually slightly outdated) here: campos-dacapo-2.7.14.tar.gz.

Only for a reference the Dacapo tar file contains the 2.7.7 version.

1.2.3   From RPM

A pretty outdated Dacapo 2.7.7 binary RPM (tested and built using Pentium 4) can be used to install Dacapo on a Pentium 4 system.

This will install pseudopotentials in /usr/share/dacapo/psp and binary executables in /usr/bin.

1.2.4   Dacapo binaries for different platforms

The page dacapo binaries list dacapo binaries for different platforms.

1.2.5   Compling the fortran source code

Compile the Dacapo source code by:

cd src
gmake <arch>           # Serial version
gmake <arch> MP=mpi    # Parallel MPI version

where <arch> is one of:

pglinux Portland Group (PGI) pgf90 compiler on Linux
intellinux Intel ifort Fortran compiler version >=9.0 and MKL on Linux
pathscale Pathscale EKOpath pathf90 compiler for AMD64 and EM64T
itanium Intel ifort compiler version >=9.0 on SGI Itanium2 Linux
intelmac Intel ifort Fortran compiler + MKL, >= 9.x on Mac OS X
g95linux GNU g95 compiler on Linux
sun_ss10 Sun sparcstation 10/20)
ibm_power3 RS/6000 power3 node
ibm_power2_wide RS/6000 wide power2 node
ibm_power2_thin RS/6000 thin/thin2 power2 node
sgi Silicon Graphics n32 ABI
alpha Compaq/Digital Alpha

It is important to use the GNU gmake command on many UNIX systems, in stead of the system's own make command which may be incompatible with our Makefile.

Please see also the Notes for installation on specific computers below.

1.2.6   Installing the pseudopotentials

Get the pseudopotentials tar-ball here: campos-dacapo-pseudopotentials-1.tar.gz.

The dacapo fortran program adds the enviroment variable DACAPOPATH to the pseudopotential filename (if the file is not found in the current working directiory). Copy all pseudopotentials to a directory and set the DACAPOPATH environment variable to this directory:

cp *.pseudo /some/directory/
setenv DACAPOPATH /some/directory/
1.2.7   Pseudopotential data files

Dacapo uses the Vanderbilt ultra-soft pseudopotentials which are read in from binary data files. It is important to compile Dacapo and NetCDF with appropriate compiler flags to enable fortran unformatted IEEE big-endian data files. Please consult this note from Vanderbilt.

Big-endian I/O is enabled in various compilers with compiler flags as follows:

  • PGI compiler: -byteswapio
  • Intel compiler: -convert big_endian
  • Pathscale compiler: -byteswapio

These flags should already be set in the Dacapo Makefile.

1.2.8   Dacapo Python interface
  1. Get the latest version of the dacapo python interface by either of these two methods:

    1. svn (the commands below will fetch the whole dacapo fortran and python source, and pseudopotentials):

      cd
      svn checkout https://USER@svn.fysik.dtu.dk/projects/dacapo/trunk dacapo
      cd dacapo/Python

    where "USER@" part has to be skipped for anonymous access.

    1. alternatively, get the tar-ball (usually slightly outdated) here: Dacapo-0.9.3.tar.gz.
  1. Install with the standard setup.py script (if you have root permission):

    python setup.py install

    If you do not have root permission use:

    python setup.py install --prefix=/some/where/in/your/path

    In this latter case you must set your PYTHONPATH environment variable, as directed by the setup.py script.

3) Alternative to step 2: Simply set the PYTHONPATH environment variable to your svn directory, i.e. ${HOME}/dacapo/Python if you followed step 1.2).

2   Running Dacapo in parallel

Dacapo can run in parallel using the MPI parallel library. You need to compile a parallel executable:

gmake <arch> MP=mpi

For getting dacapo to work in parallel with ASE you need to make a script dacapo.run, which should be executable and in your path. dacapo.run is an example of such a script. This example use the LAM/MPI and the Torque/PBS batch system.

If you do not use a batch system you can replace the line:

MACHINEFILE=$PBS_NODEFILE

with a explicit file containing the names of the nodes, one on each line:

MACHINEFILE=/your/machine/file

2.1   OpenMPI

For dacapo to run together with ASE you need a dacapo.run script in your path, that will start the correct dacapo executable. This OpenMPI dacapo.run script assumes you are running OpenMPI using the Torque/PBS batch system.

You might have to edit the location and names of the serial and parallel executable in this script, i.e the lines:

# Name of serial and parallel DACAPO executables
DACAPOEXE="dacapo_2.7.7.run"
DACAPOEXE_PAR="dacapo_2.7.7_mpi.run"

If OpenMPI is not installed under /usr you will also have to change this in the script.

Copy this script to /usr/local/bin/dacapo.run.

3   Notes for installation on specific computers

Dacapo can be built on a large number of different systems and compilers. This portability has been evolving over the years, and the supported systems are displayed by the command make in the top level source code directory. Below we give specific instructions for some systems which we actively use at our site.

If you would like to contribute new entries to the Makefile, correct errors, or add complete instructions for a new platform to the present Wiki page, please send an E-mail to support@fysik.dtu.dk.

If you would like to see how we install Dacapo etc. on our Niflheim Linux cluster and how we build RPM packages, please consult the Niflheim Wiki.

3.1   Opteron (Pathscale EKOpath compiler)

This build assumes you have the Pathscale EKOpath Fortran compiler, the OpenMPI message passing library and the ACML Math library installed. Below follows details on how to build the NetCDF and FFTW libraries needed byDdacapo.

3.1.1   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions.

Build netcdf like this:

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr FC=pathf90 FCFLAGS=-byteswapio CC=pathcc CXX=pathCC CPPFLAGS='-DNDEBUG -DpgiFortran'
make

and then install in /usr running as root:

make install

See also niflheim note on building a netcdf RPM.

3.1.2   FFTW (Fast Fourier Transform library)

Download version 2.1.5 here Build FFTW like this:

tar -zxf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure F77=pathf90 CC=pathcc CFLAGS=-O3 FFLAGS=-O3
make

and then as root:

make install

This will install FFTW in /usr/local.

3.1.3   OpenMPI

OpenMPI is an open source implementation of the MPI message passing standard. Configure OpenMPI with Pathscale Fortran and the gcc C-compiler:

./configure --prefix=/usr/local --with-tm=/usr/local FC=pathf90 F77=pathf90

The flag --with-tm=/usr/local is used only with the Torque/PBS batch system (version 2.1 or later) and enables Torque's very fast Task Manager startup of MPI processes.

3.1.4   Dacapo

Get dacapo From svn, and go to src directory:

cd dacapo/src

Set the environment variables for ACML, NETCDF and FFTW:

setenv ACML /opt/acml3.5.0/pathscale64
setenv NETCDF /usr/local
setenv FFTW /usr/local

Select the location of the MPI library which you want to use. The default is to use the MPI library installed in /usr as set by:

setenv MPIDIR /usr

Alternative locations may be specified, for example:

setenv MPIDIR /usr/local/infinipath-1.3.1

Now compile dacapo (serial and parallel):

make pathscale
make pathscale MP=mpi

Ignore warnings about type mismatch when compiling ms.F, this is due to MPI expecting pointers to integer arrays for all data types.

Now copy the compiled executables to somewhere on your path, e.g.:

cp pathscale_serial/dacapo.run /usr/local/bin/dacapo_<version>.run
cp pathscale_mpi/dacapo.run /usr/local/bin/dacapo_<version>_mpi.run

3.2   Portland Group (PGI) compiler

Here we will build using the Portland Group's PGI Workstation version 6.0-5 Fortran compiler, and a precompiled BLAS library from the ATLAS project.

3.2.1   Atlas BLAS

Take a precompiled version (3.6.0) of ATLAS that fits you system best. The rest of the installation assumes you have installed ATLAS in /usr/local/lib.

The precompiled version of ATLAS includes 2 (double) underscores for Fortran symbols, so for the rest of the Fortran compilations use the pgf90 flag -Msecond_underscore.

3.2.2   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions.

Build netcdf like this (this fails on thul: MD Aug 15 2007):

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr/local FC=pgf90 FCFLAGS='-byteswapio -Msecond_underscore' CC=pgcc CXX=pgCC CPPFLAGS='-DNDEBUG -DpgiFortran'
make

and then install as the root superuser:

make install
3.2.3   FFTW (Fast Fourier Transform library)

Download FFTW version 2.1.5 and build FFTW like this:

tar -zxf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/usr/local F77=pgf90 CC=pgcc CFLAGS=-O3 FFLAGS='-O3 -Msecond_underscore'
make

and then install as the root superuser:

make install
3.2.4   LAM-MPI

We will here use the MPI library LAM-MPI

Download the tarfile and build LAM-MPI for installation in /usr/local/lam-7.1.2-pgi and with the ssh remote-shell command, using:

tar xzvf lam-7.1.2.tar.gz
cd lam-7.1.2
setenv CPPFLAGS '-DNDEBUG -Df2cFortran'
setenv F77 pgf90
setenv FC  pgf90
setenv FFLAGS '-byteswapio -Msecond_underscore'
./configure --prefix=/usr/local/lam-7.1.2-pgi/ --with-rsh="/usr/bin/ssh -a -x" CPPFLAGS='-DNDEBUG -Df2cFortran'  F77=pgf90 FC=pgf90  FFLAGS='-byteswapio -Msecond_underscore'
make

and then install as the root superuser:

make install
3.2.5   OpenMPI

Alternatively to LAM-MPI, you may want to use OpenMPI. OpenMPI is an open source implementation of the MPI message passing standard. Configure OpenMPI version 1.1.4 with the PGI compilers:

./configure --prefix=/opt/openmpi/1.1.4 --with-tm=/usr/local FC=pgf90 F77=pgf90 CC=pgcc CXX=pgCC
make
make install  # as root

The flag --with-tm=/usr/local is used only with the Torque/PBS batch system (version 2.1 or later) and enables Torque's very fast Task Manager startup of MPI processes.

Alternatively you can build the RPM package (see https://wiki.fysik.dtu.dk/niflheim/Cluster_software#id6).

3.2.6   Dacapo

Get dacapo From svn, and go to src directory:

cd dacapo/src

Set the environment variables to use specific versions of LAPACK and BLAS. One option is to use ATLAS BLAS:

setenv BLASLAPACK '/usr/pgi/linux86/6.2/lib/liblapack.a -L/usr/local/lib -lcblas -lf77blas -latlas'

Another option is to use MKL:

setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/32 -lmkl_lapack -lmkl_ia32 -lguide -lpthread -lmkl'

Also remember to set variables for where to find netCDF, FFTW:

setenv NETCDF /usr/local
setenv FFTW /usr/local

and MPI:

setenv MPIDIR /opt/openmpi/1.2

and compile like:

make pglinux
make pglinux MP=mpi
3.2.7   Running Dacapo

If you get a runtime error from Dacapo similar to this one:

relocation error: /usr/pgi/linux86/6.2/lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

then this may possibly be a problem with the PGI compiler installation. You can use ldd dacapo.run to examine which shared libraries are needed, and if libpthread.so.0 in the /usr/pgi tree is referenced, the recommended solution is to remove the soft-link /usr/pgi/linux86/6.2/lib/libpthread.so.0 (or whatever version you have).

3.3   Intel compiler

The Intel Fortran and C/C++ compilers are installed on Intel CPUs such as Pentium-4, Xeon, Itanium etc. The Intel MKL library contains highly optimized BLAS subroutines, among many other things. It is strongly recommended that you use the latest version of the Intel compilers, since many bugs in the past have caused a number of problems for Dacapo and other software packages.

Homepages: Intel Fortran and Intel C++ Compiler version 9.1.

Manuals in PDF format are in the Intel Fortran guide and the Intel C++ guide.

3.3.1   Intel Math Kernel Library (MKL)

The Intel Math Kernel Library (MKL) Version 9.0 with MKL Manuals. A MKL User Forum for the Intel Math Kernel Library is available.

MKL 9.0 contains may of the libraries required by Dacapo: BLAS, LAPACK and FFTW (see these notes).

The FFTW library must be manually built (by the root superuser) according to the notes FFTW2.x to Intel(R) Math Kernel Library Wrappers. For example, if MKL is installed in /opt/intel/mkl/9.0, the 32-bit FFTW library is built like this:

cd /opt/intel/mkl/9.0/interfaces/fftw2xf
make lib32

Now FFTW can be linked in using these loader flags:

-L/opt/intel/mkl/9.0/lib/32 -lfftw2xf_intel
3.3.2   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions.

Build netcdf like for the Intel compiler with:

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr/local/ifort FC=ifort CC=icc CXX=icpc CPPFLAGS='-DNDEBUG -DpgiFortran'
make check

If make check fails with errors "Numeric conversion not representable" add the following flags (see http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg02387.html):

CFLAGS='-mp' FFLAGS='-mp'

and then install in /usr/local/ifort running as root:

mkdir /usr/local/ifort; make install
3.3.3   FFTW (Fast Fourier Transform library)

The Intel MKL version of FFTW should be used as discussed above.

However, if for some reason you're unable to use MKL's FFTW, you should configure the FFTW library build with:

./configure --prefix=/usr/local/ifort F77=ifort CC=icc CFLAGS=-O3 FFLAGS=-O3
3.3.4   OpenMPI

Get the latest version source tar file from OpenMPI. You must use the Intel C++ compiler version October 5, 2006 (build 44) or later, see an OpenMPI FAQ. (Note added: OpenMPI version 1.1.4 has a workaround for this bug, see the Release Notes).

Build OpenMPI with support for Torque (installation in /usr/local) and the Intel compilers using:

./configure --prefix=/usr/local/openmpi-1.1.4-intel --with-tm=/usr/local CC=icc CXX=icpc FC=ifort F77=ifort
make
make install  # as root
3.3.5   Dacapo

You must use the Intel compiler version 9.1 (or later). If you use older versions of the compiler, bugs will give you a lot of troubles.

Get dacapo From svn, and go to src directory:

cd dacapo/src

Edit the Makefile section named intellinux if you want to modify the compiler flags to generate optimal code for your particular Intel CPU, where the -x flag controls the code generation (see man ifort):

INTELLINUX_OPT = -O3 -xN

You should also select the Intel MKL library for your specific Intel CPU architecture:

# Select the appropriate Intel MKL library for your CPU (32, 64 or em64t):
MKLPATH=/opt/intel/mkl/9.0/lib/32       # Use -lmkl_ia32
MKLPATH=/opt/intel/mkl/9.0/lib/64       # Use -lmkl_ipf
MKLPATH=/opt/intel/mkl/9.0/lib/em64t    # Use -lmkl_em64t

Set the environment variables to use specific versions Netcdf, FFTW and MPI:

setenv NETCDF /usr/local/ifort
setenv MPIDIR /usr/local/openmpi-1.1.4-intel

Now build Dacapo:

make intellinux
make intellinux MP=mpi  # Parallel version
3.3.6   Running Dacapo

The default process stacksize limit may cause unexpected crashes of Dacapo at the point when all available stack space has been exhausted. Therefore you must increase the resource limits before running Dacapo:

ulimit -s unlimited        # bash shell
limit stacksize unlimited  # tcsh shell

In stead of unlimited you could enter a number such as 65000 (64 MB) so that you do not exhaust your system resources, or in case your system prohibits unlimited stack size.

You can set such commands in your personal login scripts (.bashrc or .tcshrc) or in any scripts used to run the Dacapo executable.

Dacapo: Installation (last edited 2012-09-16 17:08:32 by MarcinDulak)