PGI Guide for MPICH

This guide was created for building MPICH version 1.2.7p1 using PGI compilers.

Version Information

This guide was created for use with MPICH version 1.2.7p1 using the ch_p4 device and the PGI 2011 Release. This information is for both x64 processors running 64-bit Linux or x86 processors running 32-bit Linux.

Library Notes

MPICH is a freely available and portable implementation of MPI. Information about MPICH can be found at the MPICH home page

Obtaining the Source Code

MPICH is freely available for download from the MPICH download page.

Dependencies

There are no known dependencies.

Configuration and Set-up Information

After you have downloaded and unpacked the MPICH packages, if you are not familiar with MPICH please read the accompanying documentation found in the mpich-1.2.7p1/doc directory. For 64-bit installations, please read the list of known issues before proceeding.

To configure MPICH to use PGI compilers you must first set several environment variables and then run MPICH's configure script. The simplest method is to use the command env followed by the environment variables and configure (see example below). Some shells do not properly set the environment variables when using env. In this case, please set these variables in your environment by hand before running configure.

The exact compiler options to use will depend upon how your MPICH library will be used. For most clusters using -O2 is the best choice. However, if you are building the library for use on various 32-bit architectures, you will need to add the flag -tp px to create generic code that will run on all 32-bit systems. You will loose performance, but gain portability. For 64-bit architectures, code generated for AMD64 and Intel 64 (EM64T) will run correctly on either architecture.

If you wish to view MPI message queues when debugging your applications with PGDBG, you will need to link with a version of MPICH which has debugging enabled. To create a debug version of MPICH, compile with the -g -O0 options. You may wish to add the configure flag --enable-debug after the configure command in order to allow PGDBG to display information about the MPICH message queues.

Please modify this example as needed. Be sure to change the prefix path.

  env CFLAGS="-O2" CXXFLAGS="-O2" FFLAGS="-O2" F90FLAGS="-O2" LDFLAGS="-O2"           \
  OPTFLAGS="-O2"  CC="pgcc" CXX="pgCC"  F90="pgf90"  FC="pgf77"  CPP="pgCC -E"        \
  ./configure --prefix=/path/to/install/dir

Building MPICH

To build and install, simply run 'make' and 'make install'.

  make
  make install

After you have successfully installed the MPICH library you will need to create a 'machine' file with the list of your cluster's host names. For more information about machine files, please refer to section 4.14.1 of the MPICH manual (doc/mpichman-chp4.pdf).

Running MPICH

To test your MPICH installation, the MPICH package contains several example test programs. First make sure that the MPICH install's bin directory in in your PATH environment variable. Next change directories to the mpich-1.2.7p1/examples/test directory, then configure and run the tests.

  env CC="pgcc" CXX="pgCC" F90="pgf90" FC="pgf77" CPP="pgCC -E" ./configure        \
  --mpichpath=/path/to/install/dir/bin
  make OPTFLAGS='-O2' testing

Verifying Correctness

Results of the example tests are located in the examples/test/summary.xml file. Review this file for any errors

Using PGDBG with MPICH

The PGI graphical MPI-parallel debugger, PGDBG, is a powerful tool for assisting in the challenging task of debugging an MPI application. PGDBG is designed to work with MPICH and allows you precise control over all your MPI processes. More information about PGDBG can be found at PGDBG product page. To use PGDBG with MPICH, first copy the PGDBG debug script to your MPICH install's bin directory. Next, run mpirun using the -dbg=pgdbg option.

  mpirun -np 16 -dbg=pgdbg myapp.out

Note: You must have a PGI CDK license in order to use PGDBG with MPICH. For more information about obtaining a CDK license, please contact PGI Sales.

Known Issues and Limitations

General

-pthread

The MPICH build scripts use -pthread which can cause PGI compilers to react poorly. To work around this problem, create or edit a siterc file with these contents:

  switch -pthread is replace(-lpthread) positional(linker);

Place the file in $PGI/linux86/9.0/bin/siterc and $PGI/linux86-64/9.0/bin/siterc

32-bit Issues

longmsgs

The example test longmsgs will fail in 32-bits when using optimization level -O2 or higher. Compiling this test with -Msignextend will allow it to pass.

64-bit Issues

ssendtest and issendtest

The example tests ssendtest and issendtest will fail with the message:

  MPI_WTIME is returning 0; a working value is needed.

MPI_WTIME is working correctly, however the ch_p4 function 'p4_usclock' located in file mpid/ch_p4/p4/lib/p4_utils.c has a 64-bit porting issue where it will return 0 if not enough time has passed since initialization.

structf

The structf test will fail with the message:

  0 - MPI_ADDRESS : Address of location given to MPI_ADDRESS does not fit in Fortran integer

This error occurs due to MPICH's use of Fortran INTEGER to hold the value of an address. In Fortran, INTEGER has the default kind of 4 which is not large enough to hold a 64-bit address. Although the authors of MPICH understand this issue (See MPICH Article about 64-bit Issues), the method does not always work properly under 64-bit Linux.

A revised version of MPI_Address which correctly implements this method has been provided by Quadrics. To use this revised source, download and untar the MPICH 64-bit Patch, add "-DFIX_PTR_153_1_3800" to the CFLAGS, and run configure. Note that this patch does not completely solve MPICH's 64-bit address problem since you are still limited to accessing 2**30 bit address offsets. If you need true 64-bit addresses, you will need to migrate to MPI-2 and use MPI_Get_address.

Large Arrays

Due to buffer size limits, you are limited in the amount of actual data that can be sent with each call to MPI_SEND. It is advisable to send your array in smaller parts that are able to fit within MPI_SEND's buffer limit. Also, you can investigate the use of MPI_BSEND or MPI_IBSEND which allows you to create your own send buffer. However, you may still encounter message size limits. For arrays consisting of greater than 2 gigabytes of data, you must first compile your application using -Mlarge_arrays and then send only part of the array at a time.

MPI_ROOT not defined

If your Fortran code uses the variable MPI_ROOT, you will have to add it to the mpif.h header file.

You will need to edit /usr/pgi/linux86-64/2012/mpi/mpich/includes/mpif.h and add the lines in *bold* below for MPI_ROOT

PARAMETER (MPI_TAG_UB=80,MPI_HOST=82,MPI_IO=84)
PARAMETER (MPI_WTIME_IS_GLOBAL=86)
!
*INTEGER MPI_ROOT
PARAMETER (MPI_ROOT = (-3))*
INTEGER MPI_ANY_SOURCE
PARAMETER (MPI_ANY_SOURCE = (-2))
INTEGER MPI_ANY_TAG
PARAMETER (MPI_ANY_TAG = (-1))
Click me