June 2009
Developing Local MPI Programs with PGI Workstation
Since release 7.1 in November 2007, a version of MPICH1 has been included with PGI Workstation on Linux, and the debugger and profiler have been enabled to support MPI applications running locally with up to eight processes. With release 8.0 in November 2008, support for local MPI application debugging and profiling was extended to PGI Workstation for Windows and PGI Workstation for Mac OS X. This article describes how to build, run, debug and profile local MPI applications.
Overview
Message Passing Interface (MPI) is software used by computers connected in a cluster to communicate with one another. It is an industry standard application programming interface designed for rapid data exchange between processors in a cluster application.
Today's multicore processors bring cluster-like characteristics to workstations including the ability to host development of MPI programs. To assist developers building MPI applications, PGI now provides MPI support with PGI Workstation and PGI Server (collectively referred to in this article as PGI Workstation). On Linux, you can debug and profile MPI applications for MPICH-1, a version of which is included in your PGI Workstation installation package. PGI Workstation for Linux also supports debugging and profiling of HP-MPI, MPICH-2, MVAPICH, and OpenMPI applications, but prebuilt libraries are not included. On Microsoft Windows, if you have Microsoft's MSMPI installed you can compile, run, debug, and profile Windows MPI programs locally. On Apple Mac OS X, PGI Workstation includes a pre-configured version of OpenMPI. The following sections describe how to build, run, debug and profile MPI applications on each of the three operating systems supported by PGI Workstation.
The PGI CDK Cluster Development Kit supports MPI development , debugging and profiling on large multi-node clusters requiring remote MPI process support.
Compiling and Linking MPI Applications
The PGI compilers provide the option -Mmpi= to make building MPI applications more convenient by adding the MPI include and library directories to the compiler's include and library search paths. The following table lists the supported MPI implementations, host operating systems and -Mmpi= option:
| Implementation | Host Operating Systems | -Mmpi= Option |
| MPICH1 | Linux | -Mmpi=mpich1 |
| MVAPICH2 | Linux | -Mmpi=mpich2 |
| MVAPICH | Linux | -Mmpi=mvapich |
| HP-MPI | Linux | -Mmpi=hpmpi |
| MSMPI | Windows | -Mmpi=msmpi |
| OpenMPI | Linux, Mac OS X | N/A, see Mac OS section below |
When you build an MPI application, be sure to use these options in both the compile and link steps. Here's a simple example using MPICH-1:
% cd my_example_dir
% cp -r ${PGI}/linux86/8.0-1/EXAMPLES/MPI/mpihello .
% cd mpihello
% pgf95 -o mpihello mpihello.f -Mmpi=mpich1
% mpirun mpihello
Hello world! I'm node 0
% mpirun -np 4 mpihello
Hello world! I'm node 0
Hello world! I'm node 2
Hello world! I'm node 1
Hello world! I'm node 3
Debugging MPI Applications
PGI's PGDBG debugger supports symbolic debugging of MPI applications. The version of PGDBG that is included with PGI Workstation can debug up to eight MPI processes running on the system where PGI Workstation is installed.
For MPICH-1 programs, use the -dbg option to the mpirun command. For example, to invoke PGDBG on your mpihello program, use this command:
% mpirun -dbg=pgdbg -np 4 mpihello
For all implementations of MPI except MPICH-1, use the PGDBG command to initiate an MPI debugging session. For example, you might invoke an HP-MPI session as follows:
% pgdbg -mpi:/opt/hpmpi/bin/mpirun -np 4 hpmpihello
PGDBG can display the contents of message queues for instances of MPI that have been configured to support that feature. The version of MPICH-1 provided with PGI Workstation as well as HP-MPI and OpenMPI (Mac OS X) function properly. If you are using MPICH-2, MVAPICH, or OpenMPI on Linux, you'll need to build and configure them correctly to allow message queue dumping from within PGDBG. Some examples are available on the PGI Tips & Techniques web page. MSMPI does not support dumping of message queue contents.
Profiling MPI Applications
PGI's PGPROF performance profiler supports profiling MPI applications. Like PGDBG, the PGPROF version included with PGI Workstation supports profiling of up to eight MPI processes running on the system where PGI Workstation is installed.
To create and view a performance profile of your MPI application, first build an instrumented version of the application using the -Mprof= option and specify one of the MPI implementations listed in the table above. In addition, the -Mprof= option requires a second profiling type sub-option; valid values are func, lines, time, or hwcts. Not all sub-options are supported on all operating systems. See the PGI Tool's Guide for more information.
Here's an example for an mpich2 implementation:
% pgf95 -fast -Mprof=func,mpich2 mpihello.f
Once you have built an instrumented version of your MPI application, run it to produce the profile data. The data is contained in files named pgprof* in the working directory of the application. Invoke the PGPROF profiler in the same directory to view the data. Here's an example using our mpihello program:
% pgprof -exe mpihello
Using MPI with PGI Workstation for Windows
If you have an implementation of Microsoft's MSMPI available on the Windows system on which you have PGI Workstation installed, you can compile, run, debug, and profile MPI applications locally. MSMPI is included with Windows HPC Server 2008 and the Enterprise version of Windows Vista. The Microsoft HPC Pack 2008 SDK is freely available for use with many other versions of Windows. If you download the HPC Pack 2008 SDK, be sure to check for updates and "hot fixes" as well.
In general these instructions apply to a system where the Microsoft HPC Pack 2008 SDK is installed as a development tool, as opposed to an actual Windows HPC Server 2008 system.To build a debuggable MSMPI program using PGF95™, enter this command in the PGI Workstation command window:
PGI$ pgf95 -Mmpi=msmpi -g mpihello.f
To build a function level profiled version, enter this command:
PGI$ pgf95 -Mprof=func,msmpi mpihello.f
To run the program, use the Microsoft mpiexec utility that's included in the HPC Pack 2008 SDK. Enter this command:
PGI$ mpiexec -n 4 mpihello.exe
To debug the application with the PGDBG parallel debugger, enter this command:
PGI$ pgdbg -mpi -n 4 mpihello.exe
To profile the application with the PGPROF parallel profiler, use the same command syntax as shown above:
PGI$ pgprof -exe mpihello
Using MPI with PGI Workstation for Mac OS X
PGI Workstation for Mac OS X includes a pre-configured version of OpenMPI. Follow the instructions in the INSTALL.txt file to configure your environment. INSTALL.txt is located in your PGI installation directory, typically /opt/pgi.
Unlike other implementations of MPI, building OpenMPI applications requires using a compiler wrapper instead of compiler options. The wrapper file takes the place of the compiler command when building.
Compiling a debug version of a program on Mac OS X requires setting an option to retain the intermediate object files. There are two ways to do this. One option is to build with both the -g and -Mkeepobj options. The other method is to compile an object file first using the -c option, and then link the object file. In both the compile and link steps, include the -g option as well. Here's an example:
$ mpif90 -g -c mpihello.f
$ mpif90 -g mpihello.o -o mpihello
If your goal is to compile your program for profiling, there is no need to generate and retain object files. Build with any of the standard -Mprof= options listed above but don't include the mpi sub-option. Here's an example:
$ mpif90 -Mprof=func mpihello.f -o mpihello
To run the program…
$ mpirun -np 4 mpihello
Hello world! I'm node 0
Hello world! I'm node 1
Hello world! I'm node 2
Hello world! I'm node 3
To debug with PGDBG…
$ pgdbg -mpi:mpirun -np 4 ./mpihello
To profile with PGPROF…
$ pgprof -exe mpihello
Summary
PGI Workstation includes a full set of tools for developing MPI applications on your local desktop. The PGDBG parallel debugger and PGPROF parallel performance profiler included with PGI Workstation can debug and profile up to eight MPI processes. PGI Workstation makes MPI applications development on a local workstation a practical and cost-effective alternative to developing on a cluster.