Aalto Effort

(Samuli Hakala, Ville Havu, Jussi Enkovaara (CSC) ) We have been implementing the most performance critical C-kernels in the finite-difference mode to utilize GPUs. Implementation is done using CUDA and cuBLAS libraries when possible, Python interface to CUDA, PyCUDA is also utilized. Code supports using multiple GPUs with MPI. First tests indicate promising speedups compared to CPU-only code, and we are hoping to test larger systems (where the benefits are expected to be larger) soon. Currently, we are extending the CUDA implementation to real-time TDDFT.

Code is not in full production level yet.

Stanford/SUNCAT Effort

(Lin Li, Jun Yan, Christopher O’Grady)

We have created a GPU version of the GPAW RPA code. We observe a speedup of 40 (GPU vs. CPU+GPU) making GPUs about a factor of 5 more cost effective for this calculation. It is available in the svn branch rpa-gpu-expt.

Installation of cuda on Fedora 18 x86_64

First, cuda_5.0.35 does not support gcc version 4.7 and up, either kernel 3.7 and up, however there exist workarounds.

Proceed as follows:

  1. boot a 3.6 kernel

  2. yum -y install wget make gcc-c++ freeglut-devel libXi-devel libXmu-devel mesa-libGLU-devel

  3. configure rpmfusion-nonfree

  4. yum -y install xorg-x11-drv-nvidia-libs

  5. disable X:

    rm /etc/systemd/system/ ln -s /lib/systemd/system/ /etc/systemd/system/
  6. select a 3.6 version (see ) of kernel in /etc/grub2.cfg and make sure the linux line contains:

    nouveau.modeset=0 rd.driver.blacklist=nouveau


  7. disable kernel and nvidia updates in /etc/yum.conf:

    exclude=kernel* *nvidia*
  8. reboot

  9. download and install cuda:

    sh -override compiler

    Keep the recommended installation paths (/usr/local/cuda-5.0, …), and after the installation create a link:

    ln -s /usr/local/cuda /opt/cuda
  10. add to ~/bashrc:

    export PATH=/opt/cuda/bin:${PATH}
    export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH

    and source it.

  11. convert error into a warning in /usr/local/cuda-5.0/include/host_config.h:

    // #error — unsupported GNU version! gcc 4.7 and up are not supported!
    #warning — unsupported GNU version! gcc 4.7 and up are not supported!


  12. test:

    cd NVIDIA_CUDA-5.0_Samples/1_Utilities/deviceQuery
    make&& ./deviceQuery
  13. if X does not run the /dev/nvidia* will not be created. The solution seems to be creating them by hand (

    /sbin/modprobe nvidia
    if [ "$?" -eq 0 ]; then
      # Count the number of NVIDIA controllers found.
      NVDEVS=`lspci | grep -i NVIDIA`
      N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
      NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`
      N=`expr $N3D + $NVGA - 1`
      for i in `seq 0 $N`; do
        mknod -m 666 /dev/nvidia$i c 195 $i
      mknod -m 666 /dev/nvidiactl c 195 255
      exit 1

    Or switch back to X:

    cd /etc/systemd/system
    ln -s /lib/systemd/system/
  14. install (needed only for Aalto Effort):

    cd&& git clone
    cd ~/pycuda
    PATH=$PATH:/opt/cuda/bin LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cuda/lib64 python --update-user --boost-compiler=gcc
    /bin/mv -f ~/
    sed -i "s/boost_python-py27/boost_python/"
    sed -i 's/boost_thread/boost_thread-mt/'
    sed -i "s#'\${CUDA_ROOT}/lib', ##"
    PATH=$PATH:/opt/cuda/bin LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cuda/lib64 python install --root=~/pycuda-fc18-1

    and add to ~/.bashrc:

    export PYTHONPATH=~/pycuda-fc18-1/usr/lib64/python2.7/site-packages:${PYTHONPATH}

    May 7 2013: note that compyte has been removed from pucuda, but the source of pucuda does not reflect that. Therefore git clone and create a link under pucuda install tree. In addition,, are required by pucuda.