Differences between revisions 54 and 134 (spanning 80 versions)
Revision 54 as of 2007-01-23 17:20:48
Size: 17848
Comment:
Revision 134 as of 2012-09-16 17:08:32
Size: 32380
Editor: MarcinDulak
Comment:
Deletions are marked like this. Additions are marked like this.
Line 11: Line 11:
1) **CamposASE2** is required, see https://wiki.fysik.dtu.dk/ase The new `ASE version 3 <https://wiki.fysik.dtu.dk/ase>`_ package, unless you have a very good reason for using the old ASE version 2.

**Warning** due to the unmaintained `Numeric <http://sourceforge.net/projects/numpy/files/Old%20Numeric/>`_ package, ASE2 does not work on 64-bit systems with python version > 2.4.
If this is you case please use the `jacapo` interface from the new `ase <https://wiki.fysik.dtu.dk/ase>`_ package.

If you already have an installation of the old `ASE2 <https://wiki.fysik.dtu.dk/ase2>`_ package, it is enough just installing the `Dacapo Python interface`_.
Line 16: Line 21:
The next section describes how the fortran program, pseudopotentials and
python interface can be installed.

If you have a installation using the old CamposASE,
it is enough just installing the new python interface,
see `dacapo python interface`_.
Installation of prebuilt binaries
----------------------------------

**Preferably** install as RPM/deb. Configure the repositories `<https://wiki.fysik.dtu.dk/ase/download.html#installation-with-package-manager-on-linux>`_
and install the **dacapo** package using your distribution package manager.
On Debian/Ubuntu you may need additionally to set the `DACAPOPATH`, `DACAPOEXE_SERIAL` and `DACAPOEXE_PARALLEL` variables.

**No** further steps are necessary, please stop reading now!

If installing manually on Ubuntu see old instructions https://listserv.fysik.dtu.dk/pipermail/campos/2010-June/002635.html

If you prefer to install from tarballs
the next section describes how the fortran program, pseudopotentials and python interface can be installed.
Line 28: Line 39:
The current version of the fortran program is 2.7.7.
From svn
--------

This is the recommended method.
The commands below will fetch the whole dacapo fortran and python source, and pseudopotentials::

  cd
  svn checkout https://USER@svn.fysik.dtu.dk/projects/dacapo/trunk dacapo

where "USER@" part has to be skipped for anonymous access.

Alternatively, get the tar-ball (usually slightly outdated) here: `campos-dacapo`_.

Only for a reference the `Dacapo tar file`_ contains the ``2.7.7`` version.

.. _campos-dacapo: http://wiki.fysik.dtu.dk/dacapo-files/campos-dacapo-2.7.16.tar.gz

.. _Dacapo tar file: http://wiki.fysik.dtu.dk/dacapo-files//Dacapo-2.7.7.tar.gz
Line 33: Line 62:
A `binary RPM`_ (tested and builded using Pentium 4)
can be used to install dacapo on a Pentium 4 system.
Preferably install as RPMS: see `Configure fys yum repository <https://wiki.fysik.dtu.dk/niflheim/Cluster_software_-_RPMS#configure-fys-yum-repository>`_.

**Warning**: the section below is outdated, please ignore it.

A pretty outdated Dacapo 2.7.7 `binary RPM`_ (tested and built using Pentium 4) can be used to install Dacapo on a Pentium 4 system.
Line 38: Line 70:
This will install pseudopotentials in /usr/share/dacapo/psp and
binary executable in /usr/bin.

From Tarfile
------------

A `Dacapo tar file`_ containing fortran source code and pseudopotentials
can be used to install dacapo on a non rpm system.

Use gunzip Dacapo-2.7.7.tar.gz.
This will install pseudopotentials in ``/usr/share/dacapo/psp`` and binary executables in ``/usr/bin``.
Line 52: Line 76:
**Warning**: this section is outdated, please ignore it.
Line 62: Line 88:
   gmake <arch> [MP=mpi]    gmake <arch> # Serial version
   gmake <arch> MP=mpi # Parallel MPI version
Line 66: Line 93:
        * sun_ss10
             Sun sparcstation 10/20)
        * ibm_power3
             RS/6000 power3 node
        * ibm_power2_wide
             RS/6000 wide power2 node
        * ibm_power2_thin
             RS/6000 thin/thin2 power2 node
        * sgi
             Silicon Graphics n32 ABI
        * alpha
             Digital Alpha
        * pglinux
             Portland Group (PGI) *pgf90* compiler on Linux
        * intellinux
             Intel *ifort* Fortran compiler (previously *ifc*) version >=6.0 on Linux
        * pathscale
             Pathscale EKOpath compiler for AMD64 and EM64T
.. list-table::
  :widths: 2 5

  * - ``pglinux``
    - Portland Group (PGI) *pgf90* compiler on Linux
  * - ``intellinux``
    - Intel *ifort* Fortran compiler version >=9.0 and MKL on Linux
  * - ``pathscale``
    - Pathscale EKOpath *pathf90* compiler for AMD64 and EM64T
  * - ``itanium``
    - Intel *ifort* compiler version >=9.0 on SGI Itanium2 Linux
  * - ``intelmac``
    - Intel *ifort* Fortran compiler + MKL, >= 9.x on Mac OS X
  * - ``g95linux``
    - GNU *g95* compiler on Linux
  * - ``gfortran_fnosecond_underscore``
    - gcc-gfortran-4.1.1 compiler on CentOS
  * - ``gfortran_fsecond_underscore``
    - gcc-gfortran-4.2.1 compiler on Ubuntu
  * - ``sun_ss10``
    - Sun sparcstation 10/20)
  * - ``ibm_power3``
    - RS/6000 power3 node
  * - ``ibm_power2_wide``
    - RS/6000 wide power2 node
  * - ``ibm_power2_thin``
    - RS/6000 thin/thin2 power2 node
  * - ``sgi``
    - Silicon Graphics n32 ABI
  * - ``alpha``
    - Compaq/Digital Alpha
Line 88: Line 128:

.. _Dacapo tar file: http://dcwww.camp.dtu.dk/campos/download/Dacapo-2.7.7.tar.gz

More details should follow here...
Please see also the `Notes for installation on specific computers`_ below.
Line 97: Line 134:
The dacapo fortran program adds the enviroment
variable `DACAPOPATH` to the pseudopotential
filename
(if the file is not found in the currebt
working directiory).
Copy all pseudopotentials to a directory and set the
`DACAPOPATH` environment variable to this directory::

    cp psp/*/*/*.pseudo /some/directory/
Get the pseudopotentials tar-ball here: `campos-dacapo-pseudopotentials`_

.. _campos-dacapo-pseudopotentials: http://wiki.fysik.dtu.dk/dacapo-files/campos-dacapo-pseudopotentials-2.tar.gz

The dacapo fortran program adds the enviroment variable ``DACAPOPATH`` to the pseudopotential filename
(if the file is not found in the current working directiory).
Copy all pseudopotentials to a directory and set the ``DACAPOPATH`` environment variable to this directory::

    cp *.pseudo /some/directory/
Line 107: Line 145:

dacapo python interface
Pseudopotential data files
--------------------------

Dacapo uses the `Vanderbilt ultra-soft pseudopotentials <http://www.physics.rutgers.edu/~dhv/uspp/>`_
which are read in from binary data files.
It is important to compile Dacapo and NetCDF with appropriate compiler flags to enable
*fortran unformatted IEEE big-endian* data files.
Please consult this `note from Vanderbilt <http://www.physics.rutgers.edu/~dhv/uspp/uspp-736/Doc/FORMAT>`_.

Big-endian I/O is enabled in various compilers with compiler flags as follows:

 * **gfortran compiler:** ``-fconvert=big-endian -frecord-marker=4``

 * **Pathscale compiler:** ``-byteswapio``

 * **PGI compiler:** ``-byteswapio``

 * **Intel compiler:** ``-convert big_endian``

These flags should already be set in the Dacapo ``Makefile``.

Dacapo Python interface
Line 111: Line 169:
1) Get the latest version of the dacapo python interface.

   a) From tar file_

   b) cvs:
       cvs checkout dacapo/Python

.. _file: http://dcwww.camp.dtu.dk/campos/download/Dacapo-0.9.tar.gz


2) for case 1 unpack the tar file and::

       [home] $ cd Dacapo

   (for case 2 cd dacapo/Python)

3) and install with the standard setup.py script (if you have root permission)::

       [Python] $ python setup.py install

   If you do not have root permission use::

       [Python] $ python setup.py install --prefix=/some/where/in/your/path
**Warning** due to the unmaintained `Numeric <http://sourceforge.net/projects/numpy/files/Old%20Numeric/>`_ package, the Dacapo Python interface (as ASE 2) does not work on 64-bit systems with python version > 2.4.
If this is you case please ignore this section and use the dacapo interface from the new `ase <https://wiki.fysik.dtu.dk/ase>`_ package.

1) Get the latest version of the dacapo python interface by either of these two methods:

   a) svn (the commands below will fetch the whole dacapo fortran and python source, and pseudopotentials)::

        cd
        svn checkout https://USER@svn.fysik.dtu.dk/projects/dacapo/trunk dacapo
        cd dacapo/Python

   where "USER@" part has to be skipped for anonymous access.


   b) alternatively, get the tar-ball (usually slightly outdated) here: `Dacapo-0.9.4`_.

.. _Dacapo-0.9.4: http://wiki.fysik.dtu.dk/dacapo-files/Dacapo-0.9.4.tar.gz

2) Install with the standard setup.py script (if you have root permission)::

     python setup.py install

   If you do not have *root permission* use::

     python setup.py install --prefix=/some/where/in/your/path
Line 136: Line 196:
   as directed by the setup.py script.

4) Alternative to step 3 simply set the ``PYTHONPATH`` environment variable to
   your cvs directory.

CVS access
==========

You can access the code from CVS.
See the `Campos CVS Page`_.

After the CVS login, get the dacapo fortran source using::

      cvs -d :pserver:USERID@cvs.fysik.dtu.dk:/home/camp/CVSROOT checkout dacapo/src

Get the pseudopotentials using::

      cvs -d :pserver:USERID@cvs.fysik.dtu.dk:/home/camp/CVSROOT checkout dacapo/psp

Get the Python interface using::

      cvs -d :pserver:USERID@cvs.fysik.dtu.dk:/home/camp/CVSROOT checkout dacapo/Python


.. _Campos CVS Page: wiki:ASE:Campos_CVS
   as directed by the ``setup.py`` script.

3) Alternative to step 2: Simply set the ``PYTHONPATH`` environment variable to your ``svn`` directory,
i.e. ${HOME}/dacapo/Python if you followed step 1.2).
Line 173: Line 212:
`dacapo.run`_ is an example of such a script. This example use the
`LAM/MPI`_ and the `Torque/PBS`_ batch system.

If you do not use a batch system you can replace the line::

   MACHINEFILE=$PBS_NODEFILE

with a explicit file containing the names of the nodes, one on each line::

   MACHINEFILE=/your/machine/file

.. _dacapo.run: http://dcwww.camp.dtu.dk/campos/download/dacapo.run
The `jacapo project <https://svn.fysik.dtu.dk/projects/ase/trunk/ase/calculators/jacapo/tools>`_
provides an example of such a script. PBS, Sun grid engine, and LoadLeveler batch systems are supported.
Line 193: Line 223:
using the PBS batch system. using the *Torque/PBS* batch system.
Line 196: Line 226:
.. _ASE: https://wiki.fysik.dtu.dk/ase
Line 204: Line 235:
If OpenMPI is not installed under `/usr` you will also have to change this in the script. If OpenMPI is not installed under ``/usr`` you will also have to change this in the script.
Line 206: Line 237:
Copy this script to `/usr/local/bin/dacapo.run`. Copy this script to ``/usr/local/bin/dacapo.run``.
Line 220: Line 251:
Opteron (Pathscale EKOpath compiler)
====================================
If you would like to see how we install Dacapo etc. on our Niflheim Linux cluster and how we build RPM
packages, please consult the `Niflheim Wiki <https://wiki.fysik.dtu.dk/niflheim/Cluster_software_-_RPMS>`_.

**Note**: the most commond compilation/linking problem concerns using of incorrect underscoring convention,
when lot's of **undefined reference** errors appear, like::

  ../netcdf/gfortran_fsecond_underscore/netcdfinterface.o: In function `__netcdfinterface_MOD_local_error_handler': netcdfinterface.F:(.text+0xa3b): undefined reference to `nf_strerror__'

Please check your underscoring convention as described in this `post <https://listserv.fysik.dtu.dk/pipermail/ase-users/2009-September/000362.html>`_.

Opteron (gfortran compiler)
===========================
Line 224: Line 265:
`Pathscale EKOpath Fortran compiler <http://www.pathscale.com/ekopath.html>`_,
the OpenMPI message passing library and the
Line 227: Line 266:
Below follows details on how to build the NetCDF and FFTW libraries needed byDdacapo. Below follows details on how to build the MPI, NetCDF and FFTW libraries needed by dacapo.
Line 234: Line 273:
NetCDF version 3.6.1 or later is supported.
Line 239: Line 279:
   ./configure --prefix=/usr FC=pathf90 FCFLAGS=-byteswapio CC=pathcc CXX=pathCC CPPFLAGS='-DNDEBUG -DpgiFortran'    ./configure --prefix=/usr/local/netcdf3-3.6.1-1.gfortran \
              
FC=gfortran FCFLAGS=-O2 CC=gcc CXX=c++ CPPFLAGS='-DNDEBUG -DpgiFortran -fPIC"'
Line 242: Line 283:
and then install in /usr running as root:: and then install running as root::
Line 257: Line 298:
  ./configure F77=pathf90 CC=pathcc CFLAGS=-O3 FFLAGS=-O3   ./configure --prefix=/usr/local/fftw2-2.1.5-1.gfortran \
              F77=gfortran CC=gcc CFLAGS=-O3 FFLAGS=-O3 --enable-shared --enable-threads
Line 264: Line 306:
This will install FFTW in /usr/local. OpenMPI
-------

`OpenMPI <http://www.open-mpi.org>`_ is an open source implementation of the MPI message passing standard.
Configure OpenMPI with::

  ./configure --prefix=/usr/local/openmpi-1.2.7.gfortran --with-tm=/usr/local FC=gfortran F77=gfortran CC=gcc CXX=g++ \
              FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2

The flag ``--with-tm=/usr/local`` is used only with the Torque/PBS batch system (version 2.1 or later)
and enables Torque's very fast *Task Manager* startup of MPI processes.
Line 270: Line 322:
.. _tar-file: http://dcwww.camp.dtu.dk/campos/download/dacapo-2.7.7.tar.gz

Unpack the dacapo `tar-file`_::

     tar -xzf dacapo-2.7.7.tar.gz
     cd dacapo-2.7.7/src
Get dacapo `From svn`_, and go to `src` directory::

     cd dacapo/src
Line 277: Line 326:
Set the environment variables for ACML, NETCDF and FFTW::

     setenv ACML /opt/acml3.5.0/pathscale64
     setenv NETCDF /usr/local
     setenv FFTW /usr/local

Select the location of the MPI library which you want to use.
The default is to use the MPI library installed in ``/usr`` as set by::

     setenv MPIDIR /usr

Alternative locations may be specified, for example::

     setenv MPIDIR /usr/local/infinipath-1.3.1

Now compile dacapo (serial and parallel)::

     make pathscale
     make pathscale MP=mpi
Set the environment variables for blas/lapack, NETCDF, and FFTW, for example::

     setenv BLASLAPACK "-L/opt/acml-4.0.1/gfortran64/lib -lacml -L/opt/acml-4.0.1/gfortran64/lib -lacml -lgfortran"
     setenv NETCDF /usr/local/netcdf3-3.6.1-1.gfortran/lib64
     setenv FFTW /usr/local/fftw2-2.1.5-1.gfortran/lib64

**Warning**: if you use netcdf >= 3.6.2 (which has fortran interface in libnetcdff.a) you need to modify the BLASLAPACK variable, for example::

    setenv BLASLAPACK "/usr/local/netcdf3-3.6.2-1.gfortran/lib64 -lnetcdff -lnetcdf -L/opt/acml-4.0.1/gfortran64/lib -lacml -L/opt/acml-4.0.1/gfortran64/lib -lacml -lgfortran"

Select the location of the MPI library which you want to use::

     setenv MPIDIR /usr/local/openmpi-1.2.7.gfortran
     setenv MPI_LIBDIR ${MPIDIR}/lib64
     setenv MPI_BINDIR ${MPIDIR}/bin
     setenv MPI_INCLUDEDIR ${MPIDIR}/include

Check the underscore convention:

 - netcdf < 3.6.2::

     nm ${NETCDF}/libnetcdf.a | grep "T nf_strerror_"

 - netcdf >= 3.6.2::

     nm ${NETCDF}/libnetcdff.a | grep "T nf_strerror_"

Perform the check also on the `${FFTW}/libfftw.a` file.

Now compile dacapo with correct underscoring convention (serial and parallel)::

     make gfortran_fnosecond_underscore
     make gfortran_fnosecond_underscore MP=mpi
Line 302: Line 365:
    cp gfortran_fnosecond_underscore_serial/dacapo.run /usr/local/bin/dacapo_<version>.run
    cp gfortran_fnosecond_underscore_mpi/dacapo.run /usr/local/bin/dacapo_<version>_mpi.run


Opteron (Pathscale EKOpath compiler)
====================================

This build assumes you have the
`Pathscale EKOpath Fortran compiler <http://www.pathscale.com/ekopath.html>`_ and the
`ACML Math library <http://developer.amd.com/acml.aspx>`_ installed.
Below follows details on how to build the NetCDF and FFTW libraries needed by dacapo.

NetCDF (Network common Data Format)
-----------------------------------

Download the `NetCDF software <http://www.unidata.ucar.edu/downloads/netcdf/netcdf-3_6_1/index.jsp>`_
and read the `NetCDF installation instructions <http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install.html#Platform-Specific-Notes>`_.
NetCDF version 3.6.1 or later is supported.

Build netcdf like this::

   tar -zxf netcdf-3.6.1.tar.gz
   cd netcdf-3.6.1/src
   ./configure --prefix=/usr/local/netcdf3-3.6.1-1.pathscale \
               FC=pathf90 FCFLAGS=-byteswapio CC=pathcc CXX=pathCC CPPFLAGS='-DNDEBUG -DpgiFortran -fpic'
   make

and then install in /usr running as root::

   make install

See also `niflheim note <https://wiki.fysik.dtu.dk/niflheim/Cluster_software#netcdf-network-common-data-form>`_ on building
a netcdf RPM.

FFTW (Fast Fourier Transform library)
-------------------------------------

Download version 2.1.5 `here <http://www.fftw.org/fftw-2.1.5.tar.gz>`_
Build FFTW like this::

  tar -zxf fftw-2.1.5.tar.gz
  cd fftw-2.1.5
  ./configure --prefix=/usr/local/fftw2-2.1.5-1.pathscale \
              F77=pathf90 CC=pathcc CFLAGS=-O3 FFLAGS="-O3 --enable-shared --enable-threads"
  make

and then as root::

  make install

OpenMPI
-------

`OpenMPI <http://www.open-mpi.org>`_ is an open source implementation of the MPI message passing standard.
Configure OpenMPI with::

  ./configure --prefix=/usr/local/openmpi-1.2.7.pathscale --with-tm=/usr/local \
              FC=pathf90 F77=pathf90 CC=pathcc CXX=pathCC \
              FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2

The flag ``--with-tm=/usr/local`` is used only with the Torque/PBS batch system (version 2.1 or later)
and enables Torque's very fast *Task Manager* startup of MPI processes.


Dacapo
------

Get dacapo `From svn`_, and go to `src` directory::

     cd dacapo/src
  
Set the environment variables for blas/lapack, NETCDF, and FFTW, for example::

     setenv BLASLAPACK "-L/opt/acml-4.0.1/pathscale64/lib -lacml -L/opt/acml-4.0.1/pathscale64/lib -lacml -lgfortran"
     setenv NETCDF /usr/local/netcdf3-3.6.1-1.pathscale/lib64
     setenv FFTW /usr/local/fftw2-2.1.5-1.pathscale/lib64

Select the location of the MPI library which you want to use::

     setenv MPIDIR /usr/local/openmpi-1.2.7.pathscale
     setenv MPI_LIBDIR ${MPIDIR}/lib64
     setenv MPI_BINDIR ${MPIDIR}/bin
     setenv MPI_INCLUDEDIR ${MPIDIR}/include

Alternative locations may be specified, for example::

     setenv MPIDIR /usr/local/infinipath-2.0

Now compile dacapo (serial and parallel)::

     make pathscale
     make pathscale MP=mpi

Ignore warnings about type mismatch when compiling ``ms.F``,
this is due to MPI expecting pointers to integer arrays for all data types.
 
Now copy the compiled executables to somewhere on your path, e.g.::
Line 309: Line 470:
version 6.0-5 Fortran compiler,
and a precompiled BLAS library from the `ATLAS project <https://sourceforge.net/projects/math-atlas/>`_.

Atlas BLAS
----------

Take a precompiled version (3.6.0) of
`ATLAS <https://sourceforge.net/project/showfiles.php?group_id=23725>`_ that fits you system best.
The rest of the installation assumes you have installed ATLAS in `/usr/local/lib`.

The precompiled version of ATLAS includes 2 (double) underscores for Fortran symbols, so for the
rest of the Fortran compilations use the ``pgf90`` flag ``-Msecond_underscore``.

version 6.2-4 Fortran compiler, and the MKL library.
 
Line 329: Line 478:
NetCDF version 3.6.1 or later is supported.
Line 334: Line 484:
   ./configure --prefix=/usr/local FC=pgf90 FCFLAGS='-byteswapio -Msecond_underscore' CC=pgcc CXX=pgCC CPPFLAGS='-DNDEBUG -DpgiFortran'    ./configure --prefix=/usr/local/netcdf3-3.6.1-1.pgi \
              
FC=pgf90 FCFLAGS='-byteswapio -Msecond_underscore' CC=pgcc CXX=pgCC CPPFLAGS='-DNDEBUG -DpgiFortran'
Line 348: Line 499:
  ./configure --prefix=/usr/local F77=pgf90 CC=pgcc CFLAGS=-O3 FFLAGS='-O3 -Msecond_underscore'   ./configure --prefix=/usr/local/fftw2-2.1.5-1.pgi \
             
F77=pgf90 CC=pgcc CFLAGS=-O3 FFLAGS="-O3 --enable-shared --enable-threads -Msecond_underscore"
Line 358: Line 510:

**Warning**: These instructions are **outdated**.
Line 377: Line 531:
OpenMPI
-------

Alternatively to LAM-MPI, you may want to use OpenMPI.

`OpenMPI <http://www.open-mpi.org>`_ is an open source implementation of the MPI message passing standard.
Configure OpenMPI with::

  ./configure --prefix=/usr/local/openmpi-1.2.7.pgi --with-tm=/usr/local FC=pgf90 F77=pgf90 CC=pgcc CXX=pgCC \
              FFLAGS=-Msignextend FCFLAGS=-Msignextend \
              --with-wrapper-fflags=-Msignextend --with-wrapper-fcflags=-Msignextend"

The flag ``--with-tm=/usr/local`` is used only with the Torque/PBS batch system (version 2.1 or later)
and enables Torque's very fast *Task Manager* startup of MPI processes.
Line 380: Line 549:
Get and unpack the `Dacapo tar file`_::

   tar xzvf Dacapo-2.7.7.tar.gz
   cd Dacapo-2.7.7/src

or check the Dacapo code out from CVS::

   cvs checkout dacapo/src
   cd dacapo/src

Set the environment variables to use specific versions of LAPACK (PGI compiler version 6.2)
and ATLAS BLAS::

   setenv BLASLAPACK '/usr/pgi/linux86/6.2/lib/liblapack.a -L/usr/local/lib -lcblas -lf77blas -latlas'
   setenv NETCDF /usr/local
   setenv FFTW /usr/local
   setenv MPIDIR /usr/local/lam-7.1.2-pgi/
Get dacapo `From svn`_, and go to `src` directory::

     cd dacapo/src

Set the environment variables to use specific versions of LAPACK and BLAS::

   setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/64 -lmkl_lapack -lmkl_ia64 -lguide -lpthread -lmkl'

and for 32 bit machines::

   setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/32 -lmkl_lapack -lmkl_ia32 -lguide -lpthread -lmkl'

Also remember to set variables for where to find netCDF, FFTW::

     setenv NETCDF /usr/local/netcdf3-3.6.1-1.pgi/lib64
     setenv FFTW /usr/local/fftw2-2.1.5-1.pgi/lib64

Select the location of the MPI library which you want to use::

     setenv MPIDIR /usr/local/openmpi-1.2.7.pgi
     setenv MPI_LIBDIR ${MPIDIR}/lib64
     setenv MPI_BINDIR ${MPIDIR}/bin
     setenv MPI_INCLUDEDIR ${MPIDIR}/include
Line 402: Line 577:


Running Dacapo
--------------
Line 458: Line 637:
NetCDF version 3.6.1 or later is supported.
Line 463: Line 643:
   ./configure --prefix=/usr/local/ifort FC=ifort CC=icc CXX=icpc CPPFLAGS='-DNDEBUG -DpgiFortran'
   make

and then install in ``/usr/local/ifort`` running as root::

   make install
   ./configure --prefix=/usr/local/netcdf3-3.6.1-1.intel \
               FC=ifort CC=icc CXX=icpc CPPFLAGS='-DNDEBUG -DpgiFortran'
   make check

If make check fails with errors "Numeric conversion not representable" add the following flags
(see `<http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg02387.html>`_)::

   CFLAGS='-mp' FFLAGS='-mp'
Line 475: Line 657:
If for some reason you can't use MKL's FFTW, you can configure the FFTW library build with::

  ./configure --prefix=/usr/local/ifort F77=ifort CC=icc CFLAGS=-O3 FFLAGS=-O3
However, if for some reason you're unable to use MKL's FFTW, you should configure the FFTW library build with::

  ./configure --prefix=/usr/local/fftw2-2.1.5-1.intel \
             
F77=ifort CC=icc CFLAGS=-O3 FFLAGS="-O3 --enable-shared --enable-threads"
Line 482: Line 665:
Get the source tar file from `OpenMPI <http://www.open-mpi.org>`_.
The version now installed is 1.1.2.
Get the latest version source tar file from `OpenMPI <http://www.open-mpi.org>`_.
Line 486: Line 668:

Build OpenMPI with support for Torque (installation in ``/usr/local``) and the Intel compilers using::

  ./configure --prefix=/usr/local/openmpi-1.1.2-intel --with-tm=/usr/local CC=icc CXX=icpc FC=ifort F77=ifort
  make
(Note added: OpenMPI version 1.1.4 has a workaround for this bug, see the
`Release Notes <http://www.open-mpi.org/community/lists/announce/2007/01/0009.php>`_).

Configure OpenMPI with::

  ./configure --prefix=/usr/local/openmpi-1.2.7.intel --with-tm=/usr/local FC=ifort F77=ifort CC=icc CXX=icpc \
              FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2
              

The flag ``--with-tm=/usr/local`` is used only with the Torque/PBS batch system (version 2.1 or later)
and enables Torque's very fast *Task Manager* startup of MPI processes.
Line 498: Line 687:
Check the Dacapo code out from CVS::

   cvs checkout dacapo/src
   cd dacapo/src
Get dacapo `From svn`_, and go to `src` directory::

     cd dacapo/src
Line 504: Line 692:
particular Intel CPU, where the ``-x`` flag controls the code generation (see ``man ifort``):: particular Intel CPU, where the ``-x`` flag controls the code generation (see `man ifort <attachment:ifort.html>`_)::
Line 510: Line 698:
  # Intel MKL library (32, 64 or em64t):
  MKLPATH=/opt/intel/mkl/9.0/lib/32
   setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/64 -lmkl_lapack -lmkl_ia64 -lguide -lpthread -lmkl'

and for 32 bit machines::

   setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/32 -lmkl_lapack -lmkl_ia32 -lguide -lpthread -lmkl'
Line 515: Line 707:
   setenv NETCDF /usr/local/ifort
   setenv MPIDIR /usr/local/openmpi-1.1.2-intel
     setenv NETCDF /usr/local/netcdf3-3.6.1-1.intel/lib64
     setenv FFTW /usr/local/fftw2-2.1.5-1.intel/lib64

Select the location of the MPI library which you want to use::

     setenv MPIDIR /usr/local/openmpi-1.2.7.intel
     setenv MPI_LIBDIR ${MPIDIR}/lib64
     setenv MPI_BINDIR ${MPIDIR}/bin
     setenv MPI_INCLUDEDIR ${MPIDIR}/include

Now build Dacapo::

  make intellinux
  make intellinux MP=mpi # Parallel version
Line 522: Line 726:
Execute the following command to increase the resource limits before running Dacapo::

  ulimit -s unlimited

You can set such commands in ``/etc/profile.local`` (or similar, depending on Linux distribution)
which will be automatically executed at a (remote) login. (Thanks to Lin Zhuang <lzhuang@whu.edu.cn>).
The default process stacksize limit may cause unexpected crashes of *Dacapo* at the point when all available
stack space has been exhausted.
Often this can be seen as the txt output freezes around the line::

  WFG: Setup initial potential

Therefore you must increase the resource limits before running *Dacapo*::

  ulimit -s unlimited # bash shell
  limit stacksize unlimited # tcsh shell

**Preferably** instead of ``unlimited`` you could enter a number such as ``500000`` (500 MB, the largest required value required so far) so that you do not
exhaust your system resources, or in case your system prohibits ``unlimited`` stack size.

You can set such commands in your personal login scripts (``.bashrc`` or ``.tcshrc``)
or in any scripts used to run the *Dacapo* executable.

Performance
===========

Below the average runtime in seconds of the `memory benchmark <https://wiki.fysik.dtu.dk/gpaw/devel/benchmarks.html#memory-benchmark>`_
obtained with different compilers/libraries is given.

Benchmark is performed on a reserved node by running the following command::

  export OMP_NUM_THREADS=1
  ulimit -s 500000
  time numactl --membind=0 --physcpubind=1 python H2Al110.py --code=dacapo --runs=3 2>&1 | tee run.log

Using physical CPU number 1 with the memory node number 0 turned out to be the the fastest combination.
For description of `numa` see `NUMACTL <https://computing.llnl.gov/LCdocs/chaos/index.jsp?show=s5.2.2>`_
and `libnuma <http://oss.sgi.com/projects/libnuma/>`_.

**Note**: "fftw2" denotes `FFTW 2.1.5 <http://www.fftw.org/fftw-2.1.5.tar.gz>`_

Dual-socket dual Core AMD Opteron(tm) Processor 285/2.6 GHz/2 GB RAM per core
-----------------------------------------------------------------------------

Date: 27 May 2009

**Note**: **mkl** version **10.1.2.024** was used. Standard deviations are found below 5 sec. "**N/A**" denotes the fact
that libraries are not available, "**-**" that tests were not performed. The number of SCF steps is 16 in all cases.

============================== ======== ========== ======= ======= ====== ======
blas/lapack/fftw : compiler gfortran gfortran43 amd4.2 pathf90 ifort g95
============================== ======== ========== ======= ======= ====== ======
acml-4.2.0/acml-4.2.0/fftw2 N/A 1179.8 -- 1140.1 1147.7 N/A
acml-4.1.0/acml-4.1.0/fftw2 N/A -- -- 1102.0 -- N/A
acml-4.1.0/acml-4.1.0/mkl N/A -- -- 1051.4 -- N/A
acml-4.0.1/acml-4.0.1/fftw2 -- N/A -- 1165.7 -- N/A
blas-3.0-37/lapack-3.0-37/fftw 1484.7 1399.4 -- -- -- --
goto-1.26/acml-4.2.0/fftw2 N/A 1220.3 1288.2 1118.6 1125.6 N/A
goto-1.26/acml-4.2.0/mkl N/A 1180.9 -- -- -- N/A
goto-1.26/acml-4.0.1/fftw2 1312.2 N/A -- -- -- N/A
atlas-3.8.3/lapack-3.2.1/fftw -- 1235.3 -- -- -- 1362.7
mkl/mkl/fftw -- 1256.4 -- -- 1153.0 --
mkl/mkl/mkl -- 1209.5 -- -- -- --
============================== ======== ========== ======= ======= ====== ======

**Warning**: problems with pathscale/gfortran43: acml-4.1.0 and acml-4.2.0 have been reported.
Linking dacapo against fftw built with `pathf90 Version 3.2 2008-06-16` compiled with `-Ofast` fails.

**Note**: compilation options of fftw are important: for comparison: pathf90 Version 3.2 2008-06-16, acml-4.0.1/acml-4.0.1/fftw2, with fftw
compiled with "-O0" option (no optimizations) results in 1966.7 seconds.

================================== ============================ ============================
compiler options options for fftw
================================== ============================ ============================
gfortran 4.1.2 20080704 -O3 -funroll-all-loops -O3 -funroll-all-loops
gfortran43 4.3.2 20081007 -O3 -funroll-all-loops -O3 -funroll-all-loops
gfortran 4.2.0-amd-barcelona-rhel4 -O3 -funroll-all-loops -O3 -funroll-all-loops
pathf90 Version 3.2 2008-06-16 -Ofast -O3 -OPT:Ofast -ffast-math
ifort 11.0 083 -xHOST -O3 -ipo -no-prec-div -xHOST -O3 -ipo -no-prec-div
g95 0.91-4 gcc 4.0.3 -O3 -O3
================================== ============================ ============================

1   Download and Installation

1.1   Requirements

The new ASE version 3 package, unless you have a very good reason for using the old ASE version 2.

Warning due to the unmaintained Numeric package, ASE2 does not work on 64-bit systems with python version > 2.4. If this is you case please use the jacapo interface from the new ase package.

If you already have an installation of the old ASE2 package, it is enough just installing the Dacapo Python interface.

1.2   Installation

1.2.1   Installation of prebuilt binaries

Preferably install as RPM/deb. Configure the repositories https://wiki.fysik.dtu.dk/ase/download.html#installation-with-package-manager-on-linux and install the dacapo package using your distribution package manager. On Debian/Ubuntu you may need additionally to set the DACAPOPATH, DACAPOEXE_SERIAL and DACAPOEXE_PARALLEL variables.

No further steps are necessary, please stop reading now!

If installing manually on Ubuntu see old instructions https://listserv.fysik.dtu.dk/pipermail/campos/2010-June/002635.html

If you prefer to install from tarballs the next section describes how the fortran program, pseudopotentials and python interface can be installed.

1.2.2   Dacapo fortran program and pseudopotentials

This section describes how the fortran program and pseudopotential are installed.

1.2.3   From svn

This is the recommended method. The commands below will fetch the whole dacapo fortran and python source, and pseudopotentials:

cd
svn checkout https://USER@svn.fysik.dtu.dk/projects/dacapo/trunk dacapo

where "USER@" part has to be skipped for anonymous access.

Alternatively, get the tar-ball (usually slightly outdated) here: campos-dacapo.

Only for a reference the Dacapo tar file contains the 2.7.7 version.

1.2.4   From RPM

Preferably install as RPMS: see Configure fys yum repository.

Warning: the section below is outdated, please ignore it.

A pretty outdated Dacapo 2.7.7 binary RPM (tested and built using Pentium 4) can be used to install Dacapo on a Pentium 4 system.

This will install pseudopotentials in /usr/share/dacapo/psp and binary executables in /usr/bin.

1.2.5   Dacapo binaries for different platforms

Warning: this section is outdated, please ignore it.

The page dacapo binaries list dacapo binaries for different platforms.

1.2.6   Compling the fortran source code

Compile the Dacapo source code by:

cd src
gmake <arch>           # Serial version
gmake <arch> MP=mpi    # Parallel MPI version

where <arch> is one of:

pglinux Portland Group (PGI) pgf90 compiler on Linux
intellinux Intel ifort Fortran compiler version >=9.0 and MKL on Linux
pathscale Pathscale EKOpath pathf90 compiler for AMD64 and EM64T
itanium Intel ifort compiler version >=9.0 on SGI Itanium2 Linux
intelmac Intel ifort Fortran compiler + MKL, >= 9.x on Mac OS X
g95linux GNU g95 compiler on Linux
gfortran_fnosecond_underscore gcc-gfortran-4.1.1 compiler on CentOS
gfortran_fsecond_underscore gcc-gfortran-4.2.1 compiler on Ubuntu
sun_ss10 Sun sparcstation 10/20)
ibm_power3 RS/6000 power3 node
ibm_power2_wide RS/6000 wide power2 node
ibm_power2_thin RS/6000 thin/thin2 power2 node
sgi Silicon Graphics n32 ABI
alpha Compaq/Digital Alpha

It is important to use the GNU gmake command on many UNIX systems, in stead of the system's own make command which may be incompatible with our Makefile.

Please see also the Notes for installation on specific computers below.

1.2.7   Installing the pseudopotentials

Get the pseudopotentials tar-ball here: campos-dacapo-pseudopotentials

The dacapo fortran program adds the enviroment variable DACAPOPATH to the pseudopotential filename (if the file is not found in the current working directiory). Copy all pseudopotentials to a directory and set the DACAPOPATH environment variable to this directory:

cp *.pseudo /some/directory/
setenv DACAPOPATH /some/directory/
1.2.8   Pseudopotential data files

Dacapo uses the Vanderbilt ultra-soft pseudopotentials which are read in from binary data files. It is important to compile Dacapo and NetCDF with appropriate compiler flags to enable fortran unformatted IEEE big-endian data files. Please consult this note from Vanderbilt.

Big-endian I/O is enabled in various compilers with compiler flags as follows:

  • gfortran compiler: -fconvert=big-endian -frecord-marker=4
  • Pathscale compiler: -byteswapio
  • PGI compiler: -byteswapio
  • Intel compiler: -convert big_endian

These flags should already be set in the Dacapo Makefile.

1.2.9   Dacapo Python interface

Warning due to the unmaintained Numeric package, the Dacapo Python interface (as ASE 2) does not work on 64-bit systems with python version > 2.4. If this is you case please ignore this section and use the dacapo interface from the new ase package.

  1. Get the latest version of the dacapo python interface by either of these two methods:

    1. svn (the commands below will fetch the whole dacapo fortran and python source, and pseudopotentials):

      cd
      svn checkout https://USER@svn.fysik.dtu.dk/projects/dacapo/trunk dacapo
      cd dacapo/Python

    where "USER@" part has to be skipped for anonymous access.

    1. alternatively, get the tar-ball (usually slightly outdated) here: Dacapo-0.9.4.
  1. Install with the standard setup.py script (if you have root permission):

    python setup.py install

    If you do not have root permission use:

    python setup.py install --prefix=/some/where/in/your/path

    In this latter case you must set your PYTHONPATH environment variable, as directed by the setup.py script.

3) Alternative to step 2: Simply set the PYTHONPATH environment variable to your svn directory, i.e. ${HOME}/dacapo/Python if you followed step 1.2).

2   Running Dacapo in parallel

Dacapo can run in parallel using the MPI parallel library. You need to compile a parallel executable:

gmake <arch> MP=mpi

For getting dacapo to work in parallel with ASE you need to make a script dacapo.run, which should be executable and in your path. The jacapo project provides an example of such a script. PBS, Sun grid engine, and LoadLeveler batch systems are supported.

2.1   OpenMPI

For dacapo to run together with ASE you need a dacapo.run script in your path, that will start the correct dacapo executable. This OpenMPI dacapo.run script assumes you are running OpenMPI using the Torque/PBS batch system.

You might have to edit the location and names of the serial and parallel executable in this script, i.e the lines:

# Name of serial and parallel DACAPO executables
DACAPOEXE="dacapo_2.7.7.run"
DACAPOEXE_PAR="dacapo_2.7.7_mpi.run"

If OpenMPI is not installed under /usr you will also have to change this in the script.

Copy this script to /usr/local/bin/dacapo.run.

3   Notes for installation on specific computers

Dacapo can be built on a large number of different systems and compilers. This portability has been evolving over the years, and the supported systems are displayed by the command make in the top level source code directory. Below we give specific instructions for some systems which we actively use at our site.

If you would like to contribute new entries to the Makefile, correct errors, or add complete instructions for a new platform to the present Wiki page, please send an E-mail to support@fysik.dtu.dk.

If you would like to see how we install Dacapo etc. on our Niflheim Linux cluster and how we build RPM packages, please consult the Niflheim Wiki.

Note: the most commond compilation/linking problem concerns using of incorrect underscoring convention, when lot's of undefined reference errors appear, like:

../netcdf/gfortran_fsecond_underscore/netcdfinterface.o: In function `__netcdfinterface_MOD_local_error_handler': netcdfinterface.F:(.text+0xa3b): undefined reference to `nf_strerror__'

Please check your underscoring convention as described in this post.

3.1   Opteron (gfortran compiler)

This build assumes you have the ACML Math library installed. Below follows details on how to build the MPI, NetCDF and FFTW libraries needed by dacapo.

3.1.1   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions. NetCDF version 3.6.1 or later is supported.

Build netcdf like this:

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr/local/netcdf3-3.6.1-1.gfortran \
            FC=gfortran FCFLAGS=-O2 CC=gcc CXX=c++ CPPFLAGS='-DNDEBUG -DpgiFortran -fPIC"'
make

and then install running as root:

make install

See also niflheim note on building a netcdf RPM.

3.1.2   FFTW (Fast Fourier Transform library)

Download version 2.1.5 here Build FFTW like this:

tar -zxf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/usr/local/fftw2-2.1.5-1.gfortran \
            F77=gfortran CC=gcc CFLAGS=-O3 FFLAGS=-O3 --enable-shared --enable-threads
make

and then as root:

make install
3.1.3   OpenMPI

OpenMPI is an open source implementation of the MPI message passing standard. Configure OpenMPI with:

./configure --prefix=/usr/local/openmpi-1.2.7.gfortran --with-tm=/usr/local FC=gfortran F77=gfortran CC=gcc CXX=g++ \
            FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2

The flag --with-tm=/usr/local is used only with the Torque/PBS batch system (version 2.1 or later) and enables Torque's very fast Task Manager startup of MPI processes.

3.1.4   Dacapo

Get dacapo From svn, and go to src directory:

cd dacapo/src

Set the environment variables for blas/lapack, NETCDF, and FFTW, for example:

setenv BLASLAPACK "-L/opt/acml-4.0.1/gfortran64/lib -lacml -L/opt/acml-4.0.1/gfortran64/lib -lacml -lgfortran"
setenv NETCDF /usr/local/netcdf3-3.6.1-1.gfortran/lib64
setenv FFTW /usr/local/fftw2-2.1.5-1.gfortran/lib64

Warning: if you use netcdf >= 3.6.2 (which has fortran interface in libnetcdff.a) you need to modify the BLASLAPACK variable, for example:

setenv BLASLAPACK "/usr/local/netcdf3-3.6.2-1.gfortran/lib64 -lnetcdff -lnetcdf -L/opt/acml-4.0.1/gfortran64/lib -lacml -L/opt/acml-4.0.1/gfortran64/lib -lacml -lgfortran"

Select the location of the MPI library which you want to use:

setenv MPIDIR /usr/local/openmpi-1.2.7.gfortran
setenv MPI_LIBDIR ${MPIDIR}/lib64
setenv MPI_BINDIR ${MPIDIR}/bin
setenv MPI_INCLUDEDIR ${MPIDIR}/include

Check the underscore convention:

  • netcdf < 3.6.2:

    nm ${NETCDF}/libnetcdf.a | grep "T nf_strerror_"
  • netcdf >= 3.6.2:

    nm ${NETCDF}/libnetcdff.a | grep "T nf_strerror_"

Perform the check also on the ${FFTW}/libfftw.a file.

Now compile dacapo with correct underscoring convention (serial and parallel):

make gfortran_fnosecond_underscore
make gfortran_fnosecond_underscore MP=mpi

Ignore warnings about type mismatch when compiling ms.F, this is due to MPI expecting pointers to integer arrays for all data types.

Now copy the compiled executables to somewhere on your path, e.g.:

cp gfortran_fnosecond_underscore_serial/dacapo.run /usr/local/bin/dacapo_<version>.run
cp gfortran_fnosecond_underscore_mpi/dacapo.run /usr/local/bin/dacapo_<version>_mpi.run

3.2   Opteron (Pathscale EKOpath compiler)

This build assumes you have the Pathscale EKOpath Fortran compiler and the ACML Math library installed. Below follows details on how to build the NetCDF and FFTW libraries needed by dacapo.

3.2.1   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions. NetCDF version 3.6.1 or later is supported.

Build netcdf like this:

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr/local/netcdf3-3.6.1-1.pathscale \
            FC=pathf90 FCFLAGS=-byteswapio CC=pathcc CXX=pathCC CPPFLAGS='-DNDEBUG -DpgiFortran -fpic'
make

and then install in /usr running as root:

make install

See also niflheim note on building a netcdf RPM.

3.2.2   FFTW (Fast Fourier Transform library)

Download version 2.1.5 here Build FFTW like this:

tar -zxf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/usr/local/fftw2-2.1.5-1.pathscale \
            F77=pathf90 CC=pathcc CFLAGS=-O3 FFLAGS="-O3 --enable-shared --enable-threads"
make

and then as root:

make install
3.2.3   OpenMPI

OpenMPI is an open source implementation of the MPI message passing standard. Configure OpenMPI with:

./configure --prefix=/usr/local/openmpi-1.2.7.pathscale --with-tm=/usr/local \
            FC=pathf90 F77=pathf90 CC=pathcc CXX=pathCC \
            FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2

The flag --with-tm=/usr/local is used only with the Torque/PBS batch system (version 2.1 or later) and enables Torque's very fast Task Manager startup of MPI processes.

3.2.4   Dacapo

Get dacapo From svn, and go to src directory:

cd dacapo/src

Set the environment variables for blas/lapack, NETCDF, and FFTW, for example:

setenv BLASLAPACK "-L/opt/acml-4.0.1/pathscale64/lib -lacml -L/opt/acml-4.0.1/pathscale64/lib -lacml -lgfortran"
setenv NETCDF /usr/local/netcdf3-3.6.1-1.pathscale/lib64
setenv FFTW /usr/local/fftw2-2.1.5-1.pathscale/lib64

Select the location of the MPI library which you want to use:

setenv MPIDIR /usr/local/openmpi-1.2.7.pathscale
setenv MPI_LIBDIR ${MPIDIR}/lib64
setenv MPI_BINDIR ${MPIDIR}/bin
setenv MPI_INCLUDEDIR ${MPIDIR}/include

Alternative locations may be specified, for example:

setenv MPIDIR /usr/local/infinipath-2.0

Now compile dacapo (serial and parallel):

make pathscale
make pathscale MP=mpi

Ignore warnings about type mismatch when compiling ms.F, this is due to MPI expecting pointers to integer arrays for all data types.

Now copy the compiled executables to somewhere on your path, e.g.:

cp pathscale_serial/dacapo.run /usr/local/bin/dacapo_<version>.run
cp pathscale_mpi/dacapo.run /usr/local/bin/dacapo_<version>_mpi.run

3.3   Portland Group (PGI) compiler

Here we will build using the Portland Group's PGI Workstation version 6.2-4 Fortran compiler, and the MKL library.

3.3.1   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions. NetCDF version 3.6.1 or later is supported.

Build netcdf like this:

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr/local/netcdf3-3.6.1-1.pgi \
            FC=pgf90 FCFLAGS='-byteswapio -Msecond_underscore' CC=pgcc CXX=pgCC CPPFLAGS='-DNDEBUG -DpgiFortran'
make

and then install as the root superuser:

make install
3.3.2   FFTW (Fast Fourier Transform library)

Download FFTW version 2.1.5 and build FFTW like this:

tar -zxf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/usr/local/fftw2-2.1.5-1.pgi \
            F77=pgf90 CC=pgcc CFLAGS=-O3 FFLAGS="-O3 --enable-shared --enable-threads -Msecond_underscore"
make

and then install as the root superuser:

make install
3.3.3   LAM-MPI

Warning: These instructions are outdated.

We will here use the MPI library LAM-MPI

Download the tarfile and build LAM-MPI for installation in /usr/local/lam-7.1.2-pgi and with the ssh remote-shell command, using:

tar xzvf lam-7.1.2.tar.gz
cd lam-7.1.2
setenv CPPFLAGS '-DNDEBUG -Df2cFortran'
setenv F77 pgf90
setenv FC  pgf90
setenv FFLAGS '-byteswapio -Msecond_underscore'
./configure --prefix=/usr/local/lam-7.1.2-pgi/ --with-rsh="/usr/bin/ssh -a -x" CPPFLAGS='-DNDEBUG -Df2cFortran'  F77=pgf90 FC=pgf90  FFLAGS='-byteswapio -Msecond_underscore'
make

and then install as the root superuser:

make install
3.3.4   OpenMPI

Alternatively to LAM-MPI, you may want to use OpenMPI.

OpenMPI is an open source implementation of the MPI message passing standard. Configure OpenMPI with:

./configure --prefix=/usr/local/openmpi-1.2.7.pgi --with-tm=/usr/local FC=pgf90 F77=pgf90 CC=pgcc CXX=pgCC \
            FFLAGS=-Msignextend FCFLAGS=-Msignextend \
            --with-wrapper-fflags=-Msignextend --with-wrapper-fcflags=-Msignextend"

The flag --with-tm=/usr/local is used only with the Torque/PBS batch system (version 2.1 or later) and enables Torque's very fast Task Manager startup of MPI processes.

3.3.5   Dacapo

Get dacapo From svn, and go to src directory:

cd dacapo/src

Set the environment variables to use specific versions of LAPACK and BLAS:

setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/64 -lmkl_lapack -lmkl_ia64 -lguide -lpthread -lmkl'

and for 32 bit machines:

setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/32 -lmkl_lapack -lmkl_ia32 -lguide -lpthread -lmkl'

Also remember to set variables for where to find netCDF, FFTW:

setenv NETCDF /usr/local/netcdf3-3.6.1-1.pgi/lib64
setenv FFTW /usr/local/fftw2-2.1.5-1.pgi/lib64

Select the location of the MPI library which you want to use:

setenv MPIDIR /usr/local/openmpi-1.2.7.pgi
setenv MPI_LIBDIR ${MPIDIR}/lib64
setenv MPI_BINDIR ${MPIDIR}/bin
setenv MPI_INCLUDEDIR ${MPIDIR}/include

and compile like:

make pglinux
make pglinux MP=mpi
3.3.6   Running Dacapo

If you get a runtime error from Dacapo similar to this one:

relocation error: /usr/pgi/linux86/6.2/lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

then this may possibly be a problem with the PGI compiler installation. You can use ldd dacapo.run to examine which shared libraries are needed, and if libpthread.so.0 in the /usr/pgi tree is referenced, the recommended solution is to remove the soft-link /usr/pgi/linux86/6.2/lib/libpthread.so.0 (or whatever version you have).

3.4   Intel compiler

The Intel Fortran and C/C++ compilers are installed on Intel CPUs such as Pentium-4, Xeon, Itanium etc. The Intel MKL library contains highly optimized BLAS subroutines, among many other things. It is strongly recommended that you use the latest version of the Intel compilers, since many bugs in the past have caused a number of problems for Dacapo and other software packages.

Homepages: Intel Fortran and Intel C++ Compiler version 9.1.

Manuals in PDF format are in the Intel Fortran guide and the Intel C++ guide.

3.4.1   Intel Math Kernel Library (MKL)

The Intel Math Kernel Library (MKL) Version 9.0 with MKL Manuals. A MKL User Forum for the Intel Math Kernel Library is available.

MKL 9.0 contains may of the libraries required by Dacapo: BLAS, LAPACK and FFTW (see these notes).

The FFTW library must be manually built (by the root superuser) according to the notes FFTW2.x to Intel(R) Math Kernel Library Wrappers. For example, if MKL is installed in /opt/intel/mkl/9.0, the 32-bit FFTW library is built like this:

cd /opt/intel/mkl/9.0/interfaces/fftw2xf
make lib32

Now FFTW can be linked in using these loader flags:

-L/opt/intel/mkl/9.0/lib/32 -lfftw2xf_intel
3.4.2   NetCDF (Network common Data Format)

Download the NetCDF software and read the NetCDF installation instructions. NetCDF version 3.6.1 or later is supported.

Build netcdf like for the Intel compiler with:

tar -zxf netcdf-3.6.1.tar.gz
cd netcdf-3.6.1/src
./configure --prefix=/usr/local/netcdf3-3.6.1-1.intel \
            FC=ifort CC=icc CXX=icpc CPPFLAGS='-DNDEBUG -DpgiFortran'
make check

If make check fails with errors "Numeric conversion not representable" add the following flags (see http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg02387.html):

CFLAGS='-mp' FFLAGS='-mp'
3.4.3   FFTW (Fast Fourier Transform library)

The Intel MKL version of FFTW should be used as discussed above.

However, if for some reason you're unable to use MKL's FFTW, you should configure the FFTW library build with:

./configure --prefix=/usr/local/fftw2-2.1.5-1.intel \
            F77=ifort CC=icc CFLAGS=-O3 FFLAGS="-O3 --enable-shared --enable-threads"
3.4.4   OpenMPI

Get the latest version source tar file from OpenMPI. You must use the Intel C++ compiler version October 5, 2006 (build 44) or later, see an OpenMPI FAQ. (Note added: OpenMPI version 1.1.4 has a workaround for this bug, see the Release Notes).

Configure OpenMPI with:

./configure --prefix=/usr/local/openmpi-1.2.7.intel --with-tm=/usr/local FC=ifort F77=ifort CC=icc CXX=icpc \
            FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2

The flag --with-tm=/usr/local is used only with the Torque/PBS batch system (version 2.1 or later) and enables Torque's very fast Task Manager startup of MPI processes.

3.4.5   Dacapo

You must use the Intel compiler version 9.1 (or later). If you use older versions of the compiler, bugs will give you a lot of troubles.

Get dacapo From svn, and go to src directory:

cd dacapo/src

Edit the Makefile section named intellinux if you want to modify the compiler flags to generate optimal code for your particular Intel CPU, where the -x flag controls the code generation (see man ifort):

INTELLINUX_OPT = -O3 -xN

You should also select the Intel MKL library for your specific Intel CPU architecture:

setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/64 -lmkl_lapack -lmkl_ia64 -lguide -lpthread -lmkl'

and for 32 bit machines:

setenv BLASLAPACK '-L/opt/intel/mkl/9.0/lib/32 -lmkl_lapack -lmkl_ia32 -lguide -lpthread -lmkl'

Set the environment variables to use specific versions Netcdf, FFTW and MPI:

setenv NETCDF /usr/local/netcdf3-3.6.1-1.intel/lib64
setenv FFTW /usr/local/fftw2-2.1.5-1.intel/lib64

Select the location of the MPI library which you want to use:

setenv MPIDIR /usr/local/openmpi-1.2.7.intel
setenv MPI_LIBDIR ${MPIDIR}/lib64
setenv MPI_BINDIR ${MPIDIR}/bin
setenv MPI_INCLUDEDIR ${MPIDIR}/include

Now build Dacapo:

make intellinux
make intellinux MP=mpi  # Parallel version
3.4.6   Running Dacapo

The default process stacksize limit may cause unexpected crashes of Dacapo at the point when all available stack space has been exhausted. Often this can be seen as the txt output freezes around the line:

WFG: Setup initial potential

Therefore you must increase the resource limits before running Dacapo:

ulimit -s unlimited        # bash shell
limit stacksize unlimited  # tcsh shell

Preferably instead of unlimited you could enter a number such as 500000 (500 MB, the largest required value required so far) so that you do not exhaust your system resources, or in case your system prohibits unlimited stack size.

You can set such commands in your personal login scripts (.bashrc or .tcshrc) or in any scripts used to run the Dacapo executable.

3.5   Performance

Below the average runtime in seconds of the memory benchmark obtained with different compilers/libraries is given.

Benchmark is performed on a reserved node by running the following command:

export OMP_NUM_THREADS=1
ulimit -s 500000
time numactl --membind=0 --physcpubind=1 python H2Al110.py --code=dacapo --runs=3 2>&1 | tee run.log

Using physical CPU number 1 with the memory node number 0 turned out to be the the fastest combination. For description of numa see NUMACTL and libnuma.

Note: "fftw2" denotes FFTW 2.1.5

3.5.1   Dual-socket dual Core AMD Opteron(tm) Processor 285/2.6 GHz/2 GB RAM per core

Date: 27 May 2009

Note: mkl version 10.1.2.024 was used. Standard deviations are found below 5 sec. "N/A" denotes the fact that libraries are not available, "-" that tests were not performed. The number of SCF steps is 16 in all cases.

blas/lapack/fftw : compiler gfortran gfortran43 amd4.2 pathf90 ifort g95
acml-4.2.0/acml-4.2.0/fftw2 N/A 1179.8 -- 1140.1 1147.7 N/A
acml-4.1.0/acml-4.1.0/fftw2 N/A -- -- 1102.0 -- N/A
acml-4.1.0/acml-4.1.0/mkl N/A -- -- 1051.4 -- N/A
acml-4.0.1/acml-4.0.1/fftw2 -- N/A -- 1165.7 -- N/A
blas-3.0-37/lapack-3.0-37/fftw 1484.7 1399.4 -- -- -- --
goto-1.26/acml-4.2.0/fftw2 N/A 1220.3 1288.2 1118.6 1125.6 N/A
goto-1.26/acml-4.2.0/mkl N/A 1180.9 -- -- -- N/A
goto-1.26/acml-4.0.1/fftw2 1312.2 N/A -- -- -- N/A
atlas-3.8.3/lapack-3.2.1/fftw -- 1235.3 -- -- -- 1362.7
mkl/mkl/fftw -- 1256.4 -- -- 1153.0 --
mkl/mkl/mkl -- 1209.5 -- -- -- --

Warning: problems with pathscale/gfortran43: acml-4.1.0 and acml-4.2.0 have been reported. Linking dacapo against fftw built with pathf90 Version 3.2 2008-06-16 compiled with -Ofast fails.

Note: compilation options of fftw are important: for comparison: pathf90 Version 3.2 2008-06-16, acml-4.0.1/acml-4.0.1/fftw2, with fftw compiled with "-O0" option (no optimizations) results in 1966.7 seconds.

compiler options options for fftw
gfortran 4.1.2 20080704 -O3 -funroll-all-loops -O3 -funroll-all-loops
gfortran43 4.3.2 20081007 -O3 -funroll-all-loops -O3 -funroll-all-loops
gfortran 4.2.0-amd-barcelona-rhel4 -O3 -funroll-all-loops -O3 -funroll-all-loops
pathf90 Version 3.2 2008-06-16 -Ofast -O3 -OPT:Ofast -ffast-math
ifort 11.0 083 -xHOST -O3 -ipo -no-prec-div -xHOST -O3 -ipo -no-prec-div
g95 0.91-4 gcc 4.0.3 -O3 -O3

Dacapo: Installation (last edited 2012-09-16 17:08:32 by MarcinDulak)