PGI Guide to MVAPICH & MVAPICH2

This guide is intended to help build the MVAPICH and MVAPICH2 libraries using the PGI 2011 compilers.

Version
Library Notes
Source Information
Dependencies
Building and Installing MVAPICH
Building and Installing MVAPICH2
Known Issues

Version Information

This guide was created for the 1.2rc1 release of the MVAPICH libraries and the 1.8a1p1 release of the MVAPICH2 libraries. This information is for both x64 processors running 64-bit Linux or x86 processors running 32-bit Linux.

Library Notes

MVAPICH and MVAPICH2 are freely available from the MVAPICH web site.

From the webpage

MVAPICH and MVAPICH2 are high-performance implementations of the Message Passing Interface (MPI) standard which run over InfiniBand interconnects.

The from MVAPICH webpage:

"Welcome to this web page related to `MPI over InfiniBand, 10GigE/iWARP and RDMA over Converged Ethernet (RoCE)' project, lead by Network-Based Computing Laboratory (NBCL) of the Ohio State University. MVAPICH/MVAPICH2 software delivers best performance, scalability and fault tolerance for high-end computing systems and servers using InfiniBand, 10GigE/iWARP and RoCE networking technologies. This software is being used by more than 1,820 organizations world-wide in 65 countries (Current Users) to extract the potential of these emerging networking technologies for modern systems. As of December '11, more than 88,000 downloads have taken place from this project's site. This software is also being distributed by many InfiniBand, 10GigE/iWARP and RoCE vendors in their software distributions. MVAPICH and MVAPICH2 are also available with Open Fabrics Enterprise Distribution (OFED) stack. MVAPICH/MVAPICH2 software is powering several supercomputers in the TOP 500 list."

Obtaining the Source Code

Source code can be downloaded by following the link at the MVAPICH home page

Dependencies

MVAPICH and MVAPICH2 require that you already have the InfiniBand libraries installed on your build system.

Building and Installing MVAPICH

Untar the MVAPICH package:

  tar -xvzf mvapich-1.2rc1.tar.gz
  cd mvapich-1.2rc1

To use the default configuration edit the file make.mvapich.gen2 and set the "CC" variable to "pgcc". The script also has the ability to automatically detect whether the platform has support for RDMAoE and configure the library to use it accordingly. For RDMAoE functionality to work properly, a version of OFED from the OFED-1.5-RDMAoE branch must be installed on all the systems.
```
  ./configure --prefix=/usr/local/mvapich >&  configure.log
```
There are some common configuration options to the default configuration that might be of interest.
```
  ./configure --enable-sharedlib       # enables shared libraries
  ./configure --enable-sharedlib --enable-debug  # enables debugging for TotalView
  ./configure --with-romio --with-file-system=lustre+nfs+pvfs2 # enables optimized drivers for 
  the listed filesystems 
```
Build the code:
```
  make >& make.log   
```
Install the code:
```
  sudo su
  make install
```

To build MVAPICH with OpenFabrics/Gen2 Hybrid Mode (Gen2-Hybrid) edit the file make.mvapich.hybrid and set the "CC" variable to "pgcc":

  ./configure --prefix=/usr/local/mvapich >&  configure.log

There are some common configuration options to the default configuration that might be of interest:

  ./configure --enable-sharedlib       # enables shared libraries
  ./configure --enable-sharedlib --enable-debug  # enables debugging for TotalView

Build the code:

  make >& make.log

Install the code:

  sudo su
  make install

After you have successfully installed the MVAPICH library you can use mpirun_rsh -np <number> -hostfile ~/.hosts my_MPI_example to launch your MPI jobs over you InfiniBand fabric.

Building and Installing MVAPICH2

Untar the MVAPICH2 package:

  tar -xvzf mvapich2-1.8a1p1.tar.gz
  cd mvapich2-1.8a1p1

The simplest way to get started with MVAPICH2 is simply:

  env CC=pgcc ./configure --prefix=/usr/local/mvapich2  >& configure.log

There are some common configuration options to the default configuration that might be of interest.

OpenFabrics (OFA) IB/iWARP/RoCE with the CH3 channel is the default interface on Linux. It 
can be explicitly selected by configuring with:

  ./configure --with-device=ch3:mrail --with-rdma=gen2

In order to use shared libraries configure as follows:

  ./configure --with-device=ch3:mrail --with-rdma=gen2 --enable-shared

To enable use of the TotalView debugger, the library needs to be configured in the following 
manner:

  ./configure --with-device=ch3:mrail --with-rdma=gen2 --enable-shared  --enable-g=dbg      \
  --enable-debuginfo

Configuration Options for OpenFabrics IB/iWARP/RoCE

    Configuring with Shared Libraries
        Default: Disabled
        Enable: --enable-shared
    Configuring with TotalView support
        Default: Disabled
        Enable: --enable-g=dbg --enable-shared --enable-debuginfo
    Path to OpenFabrics Header Files
        Default: Your PATH
        Specify: --with-ib-include=path
    Path to OpenFabrics Libraries
        Default: The systems search path for libraries.
        Specify: --with-ib-libpath=path
    Support for Hybrid UD-RC/XRC transports
        Default: Disabled
        Enable: --enable-hybrid
    Support for RDMA CM
        Default: enabled, except when BLCR support is enabled
        Disable: --disable-rdma-cm
    Support for RoCE
        Default: enabled
        For RoCE functionality to work properly, a version of OFED from the OFED-1.5-RoCE 
        branch must be installed on all the systems.
    Registration Cache
        Default: enabled
        Disable: --disable-registration-cache
    ADIO driver for Lustre:
        When compiled with this support, MVAPICH2 will use the optimized driver for 
        Lustre. In order to enable this feature, the flag
        --enable-romio --with-file-system=lustre
        should be passed to configure (--enable-romio is optional as it is enabled by 
        default). You can add support for more file systems using
        --enable-romio --with-file-system=lustre+nfs+pvfs2
    LiMIC2 Support
        Default: disabled
        Enable:
        --with-limic2[=]
        --with-limic2-include=
        --with-limic2-libpath=
    Header Caching
        Default: enabled
        Disable: --disable-header-caching
    Checkpoint/Restart
        Option name: --enable-ckpt
        Require: Berkeley Lab Checkpoint/Restart (BLCR)
        Default: disabled

    The Berkeley Lab Checkpoint/Restart (BLCR) installation is automatically detected if 
    installed in the standard location. To specify an alternative path to the BLCR 
     installation, you can either use:
    --with-blcr=
    or
    --with-blcr-include=
    --with-blcr-libpath=

    Process Migration
        Option name: --enable-ckpt-migration
        Automatically enable Checkpoint/Restart
        Require: Fault Tolerance Backplane (FTB)
        Default: disabled

    The Fault Tolerance Backplane (FTB) installation is automatically detected if installed 
    in the standard location. To specify an alternative path to the FTB installation, you can 
    either use:
    --with-ftb=
    or
    --with-ftb-include=
    --with-ftb-libpath=

    Checkpoint Aggregation
        Option name: --enable-ckpt-aggregation or --disable-ckpt-aggregation
        Automatically enable Checkpoint/Restart
        Require: Filesystem in Userspace (FUSE)
        Default: enabled (if Checkpoint/Restart enabled and FUSE is present)

    The Filesystem in Userspace (FUSE) installation is automatically detected if installed 
    in the standard location. To specify an alternative path to the FUSE installation, you 
     can either use:
    --with-fuse=
    or
    --with-fuse-include=
    --with-fuse-libpath=

    eXtended Reliable Connection
        Default: disabled
        Enable: --enable-xrc
    HWLOC Support (Affinity)
        Default: enabled
        Disable: --without-hwloc
    Support for 64K or greater number of cores
        Default: 64K or lower number of cores
        Enable: --with-ch3-rank-bits=32

Build the code:
```
  make >& make.log   
```
Install the code:
```
  sudo su
  make install
```
After you have successfully installed the MVAPICH2 library you can use mpirun_rsh -np <number> -hostfile ~/.hosts my_MPI_example to launch your MPI jobs over you InfiniBand fabric.

Known Issues and Limitations

For MVAPICH2 build issues with PGI 12.2, in the two files listed below change

#if !defined(_GNU_SOURCE)

#if !defined(_GNU_SOURCE) || !defined(__cplusplus)

/opt/pgi/linux86-64/12.2/include/float.h
/opt/pgi/linux86-64/12.2/include/limits.h

Click me