PGI Guide to MVAPICH & MVAPICH2

This guide is intended to help build the MVAPICH and MVAPICH2 libraries using the PGI 2011 compilers.

Version Information

This guide was created for the 1.2rc1 release of the MVAPICH libraries and the 1.8a1p1 release of the MVAPICH2 libraries. This information is for both x64 processors running 64-bit Linux or x86 processors running 32-bit Linux.

Library Notes

MVAPICH and MVAPICH2 are freely available from the MVAPICH web site.

From the webpage

MVAPICH and MVAPICH2 are high-performance implementations of the Message Passing Interface (MPI) standard which run over InfiniBand interconnects.

The from MVAPICH webpage:

"Welcome to this web page related to `MPI over InfiniBand, 10GigE/iWARP and RDMA over Converged Ethernet (RoCE)' project, lead by Network-Based Computing Laboratory (NBCL) of the Ohio State University. MVAPICH/MVAPICH2 software delivers best performance, scalability and fault tolerance for high-end computing systems and servers using InfiniBand, 10GigE/iWARP and RoCE networking technologies. This software is being used by more than 1,820 organizations world-wide in 65 countries (Current Users) to extract the potential of these emerging networking technologies for modern systems. As of December '11, more than 88,000 downloads have taken place from this project's site. This software is also being distributed by many InfiniBand, 10GigE/iWARP and RoCE vendors in their software distributions. MVAPICH and MVAPICH2 are also available with Open Fabrics Enterprise Distribution (OFED) stack. MVAPICH/MVAPICH2 software is powering several supercomputers in the TOP 500 list."

Obtaining the Source Code

Source code can be downloaded by following the link at the MVAPICH home page

Dependencies

MVAPICH and MVAPICH2 require that you already have the InfiniBand libraries installed on your build system.

Building and Installing MVAPICH

  1. Untar the MVAPICH package:

      tar -xvzf mvapich-1.2rc1.tar.gz
      cd mvapich-1.2rc1
    
  2. To use the default configuration edit the file make.mvapich.gen2 and set the "CC" variable to "pgcc". The script also has the ability to automatically detect whether the platform has support for RDMAoE and configure the library to use it accordingly. For RDMAoE functionality to work properly, a version of OFED from the OFED-1.5-RDMAoE branch must be installed on all the systems.

      ./configure --prefix=/usr/local/mvapich >&  configure.log
    

    There are some common configuration options to the default configuration that might be of interest.

      ./configure --enable-sharedlib       # enables shared libraries
      ./configure --enable-sharedlib --enable-debug  # enables debugging for TotalView
      ./configure --with-romio --with-file-system=lustre+nfs+pvfs2 # enables optimized drivers for 
      the listed filesystems 
    

    Build the code:

      make >& make.log   
    

    Install the code:

      sudo su
      make install
    
  3. To build MVAPICH with OpenFabrics/Gen2 Hybrid Mode (Gen2-Hybrid) edit the file make.mvapich.hybrid and set the "CC" variable to "pgcc":

      ./configure --prefix=/usr/local/mvapich >&  configure.log
    

    There are some common configuration options to the default configuration that might be of interest:

      ./configure --enable-sharedlib       # enables shared libraries
      ./configure --enable-sharedlib --enable-debug  # enables debugging for TotalView
    

    Build the code:

      make >& make.log   
    

    Install the code:

      sudo su
      make install
    
  4. After you have successfully installed the MVAPICH library you can use mpirun_rsh -np <number> -hostfile ~/.hosts my_MPI_example to launch your MPI jobs over you InfiniBand fabric.

Building and Installing MVAPICH2

  1. Untar the MVAPICH2 package:

      tar -xvzf mvapich2-1.8a1p1.tar.gz
      cd mvapich2-1.8a1p1
    
  2. The simplest way to get started with MVAPICH2 is simply:

      env CC=pgcc ./configure --prefix=/usr/local/mvapich2  >& configure.log
    

    There are some common configuration options to the default configuration that might be of interest.

    OpenFabrics (OFA) IB/iWARP/RoCE with the CH3 channel is the default interface on Linux. It 
    can be explicitly selected by configuring with:
    
      ./configure --with-device=ch3:mrail --with-rdma=gen2
    
    In order to use shared libraries configure as follows:
    
      ./configure --with-device=ch3:mrail --with-rdma=gen2 --enable-shared
    
    To enable use of the TotalView debugger, the library needs to be configured in the following 
    manner:
    
      ./configure --with-device=ch3:mrail --with-rdma=gen2 --enable-shared  --enable-g=dbg      \
      --enable-debuginfo
    
    Configuration Options for OpenFabrics IB/iWARP/RoCE
    
        Configuring with Shared Libraries
            Default: Disabled
            Enable: --enable-shared
        Configuring with TotalView support
            Default: Disabled
            Enable: --enable-g=dbg --enable-shared --enable-debuginfo
        Path to OpenFabrics Header Files
            Default: Your PATH
            Specify: --with-ib-include=path
        Path to OpenFabrics Libraries
            Default: The systems search path for libraries.
            Specify: --with-ib-libpath=path
        Support for Hybrid UD-RC/XRC transports
            Default: Disabled
            Enable: --enable-hybrid
        Support for RDMA CM
            Default: enabled, except when BLCR support is enabled
            Disable: --disable-rdma-cm
        Support for RoCE
            Default: enabled
            For RoCE functionality to work properly, a version of OFED from the OFED-1.5-RoCE 
            branch must be installed on all the systems.
        Registration Cache
            Default: enabled
            Disable: --disable-registration-cache
        ADIO driver for Lustre:
            When compiled with this support, MVAPICH2 will use the optimized driver for 
            Lustre. In order to enable this feature, the flag
            --enable-romio --with-file-system=lustre
            should be passed to configure (--enable-romio is optional as it is enabled by 
            default). You can add support for more file systems using
            --enable-romio --with-file-system=lustre+nfs+pvfs2
        LiMIC2 Support
            Default: disabled
            Enable:
            --with-limic2[=]
            --with-limic2-include=
            --with-limic2-libpath=
        Header Caching
            Default: enabled
            Disable: --disable-header-caching
        Checkpoint/Restart
            Option name: --enable-ckpt
            Require: Berkeley Lab Checkpoint/Restart (BLCR)
            Default: disabled
    
        The Berkeley Lab Checkpoint/Restart (BLCR) installation is automatically detected if 
        installed in the standard location. To specify an alternative path to the BLCR 
         installation, you can either use:
        --with-blcr=
        or
        --with-blcr-include=
        --with-blcr-libpath=
    
        Process Migration
            Option name: --enable-ckpt-migration
            Automatically enable Checkpoint/Restart
            Require: Fault Tolerance Backplane (FTB)
            Default: disabled
    
        The Fault Tolerance Backplane (FTB) installation is automatically detected if installed 
        in the standard location. To specify an alternative path to the FTB installation, you can 
        either use:
        --with-ftb=
        or
        --with-ftb-include=
        --with-ftb-libpath=
    
        Checkpoint Aggregation
            Option name: --enable-ckpt-aggregation or --disable-ckpt-aggregation
            Automatically enable Checkpoint/Restart
            Require: Filesystem in Userspace (FUSE)
            Default: enabled (if Checkpoint/Restart enabled and FUSE is present)
    
        The Filesystem in Userspace (FUSE) installation is automatically detected if installed 
        in the standard location. To specify an alternative path to the FUSE installation, you 
         can either use:
        --with-fuse=
        or
        --with-fuse-include=
        --with-fuse-libpath=
    
        eXtended Reliable Connection
            Default: disabled
            Enable: --enable-xrc
        HWLOC Support (Affinity)
            Default: enabled
            Disable: --without-hwloc
        Support for 64K or greater number of cores
            Default: 64K or lower number of cores
            Enable: --with-ch3-rank-bits=32
    
    
  3. Build the code:

      make >& make.log   
    
  4. Install the code:

      sudo su
      make install
    

    After you have successfully installed the MVAPICH2 library you can use mpirun_rsh -np <number> -hostfile ~/.hosts my_MPI_example to launch your MPI jobs over you InfiniBand fabric.

Known Issues and Limitations

For MVAPICH2 build issues with PGI 12.2, in the two files listed below change

#if !defined(_GNU_SOURCE)

to

#if !defined(_GNU_SOURCE) || !defined(__cplusplus)

/opt/pgi/linux86-64/12.2/include/float.h
/opt/pgi/linux86-64/12.2/include/limits.h

Click me