August 2009
Using Microsoft MPI with PGI Visual Fortran
If you've ever had any interest in or even just basic curiosity about running an MPI application on a Windows system, you should read this article. With the 9.0 release, PGI Visual Fortran (PVF) and PGI Workstation for Windows fully support building, running and debugging MPI applications on systems ranging from single core laptops to Windows HPC Server 2008 clusters.
We'll start by building an MSMPI application from within a PVF project. We'll then walk through the steps of running and debugging it locally and on a cluster.
Never used Visual Studio before? Download PVF with the VS Shell and take it for a spin today with a free 15 day evaluation license.
Build an MSMPI Application
Let's create a simple MSMPI application in PVF. Open PGI Visual Fortran and select File | New | Project. In the New Project dialog, select the Empty Project from among the PGI Visual Fortran project types. Name the project "BasicMPI" and click OK.
Right-click in the Solution Explorer (where your project appears) and select Add | New Item to add a file to the project. In the Add New File dialog, select a Fortran 90 file, name it prog.f90 and click OK.
Now you're ready to add some source code. Here's a simple MPI program:
program BasicMPI include 'mpif.h' integer iStatus, iRank, iRet call MPI_Init(iRet) call MPI_Initialized(iStatus, iRet) call MPI_Comm_rank(MPI_COMM_WORLD, iRank, iRet) print *, "Rank:", iRank call MPI_Finalize(iRet) end
Add the source to the source file and compile it using Build | Build Solution.
If you've been following along exactly, you'll get a build error like:
C:\testing\BasicMPI\BasicMPI\prog.f90(3) : error S0017 : Unable to open include
file: mpif.h 0 inform, 0 warnings, 1 severes, 0 fatal for basicmpi BasicMPI build failed.
The compiler needs to know that we're using MSMPI. With your project selected in the Solution Explorer, open your solution's property pages via View | Property Pages and navigate to the Fortran | Language page. Find the MPI property and set its value to Microsoft MPI.
Try building again. If you've already installed MSMPI on your system, your application should compile without errors. If you haven't installed MSMPI yet, you can download the HPC Pack 2008 SDK directly from Microsoft. Installing this package provides the MSMPI headers, libraries and tools you'll need to build your application.
Run and Debug an MSMPI Application Locally
Run your application using Debug | Start Without Debugging or Ctrl+F5. You should see the following output in a command window:
Rank: 0 Press any key to continue . . .
You just ran the application in serial mode. Because this is an MPI application, though, we can also run it using multiple processes. The key to PVF's MPI modes lies in the Debugging property page. The value you select for the MPI Debugging property on this page controls the way that PVF launches your program.
Open your solution's property pages to see the properties listed on the Debugging page. The MPI Debugging property is set to Disabled by default; this configuration produces a serial run.
Change the MPI Debugging property to Local. Additional properties will then appear on the page; these properties are specific to running MPI applications locally (if you don't see the new properties right away, click Apply).
Let's run the application with four processes. In the Number of Processes property, enter 4 and click OK.
You don't need to rebuild your program, just run it without debugging. The output window will now show evidence of a multi-process run:
Rank: 2 Rank: 1 Rank: 0 Rank: 3 Press any key to continue . . .
Debugging the application is also straightforward. Let's set a breakpoint on the print line:
To start debugging, select Debug | Start Debugging or press F5. Execution stops on the breakpoint. Open the Processes window (Debug | Windows | Processes) to view the processes under debug:
View the Breakpoints window (Debug | Windows | Breakpoints) and note that the breakpoint you set has been bound on a per-process basis:
Run and Debug an MSMPI Application on a Cluster
If you are working on a Windows HPC Server 2008 Cluster, you can run and debug your MSMPI application on both the head and compute nodes. PVF launches MSMPI applications using Microsoft's HPC Job Manager, which you can then use to track the status of your jobs.To continue the hands-on portion in the remainder of this article, you'll need a Microsoft HPC Server 2008 Cluster and a PGI CDK license. Trial CDK licenses are available; contact sales@pgroup.com.
The first step to running your application on a cluster is to change the MPI Debugging property to Cluster (you'll need the CDK license even for this step). Additional properties will then appear on the page; these properties are specific to running MPI applications on a cluster.
You can only specify as many cores in the Number of Cores property as you have available on the cluster. This restriction is imposed by the Job Manager. Use the cluscfg tool to find out information about your cluster including the number of cores. For example:
CMD> cluscfg view
Cluster name : hpc-head
Version : 2.0.1551.0
Total number of nodes : 7
Ready nodes : 7
Offline nodes : 0
Draining nodes : 0
Unreachable nodes : 0
Total number of cores : 11
Busy cores : 0
Idle cores : 11
Offline cores : 0
Total number of jobs : 2
Configuring jobs : 0
Submitted jobs : 0
Validating jobs : 0
Queued jobs : 0
Running jobs : 0
Finishing jobs : 0
Finished jobs : 1
Canceling jobs : 0
Canceled jobs : 0
Failed jobs : 1
Total number of tasks : 2
Configuring tasks : 0
Submitted tasks : 0
Queued tasks : 0
Running tasks : 0
Finished tasks : 1
Canceling tasks : 0
Canceled tasks : 0
Failed tasks : 1
Set the Number of Cores property accordingly. We have eleven cores available on seven nodes; we'll use eight cores in our example.
We need to make sure of one more thing before we'll be ready to run. For an application to run on a cluster, it must be located in a directory that has been designated as shared. Use Windows Explorer to select the folder to share and right-click on it. Select the Share option to open the File Sharing dialog box. Click the Share button. Your directory will now have a Sharing icon associated with it:
Now you are ready to run your program on the cluster. Try running without debugging (Ctrl+F5). You'll notice the command window no longer shows program output. Instead, you'll see a launch confirmation like the following:
Job has been submitted. ID: 362. Press any key to continue . . .
It's a good idea to verify that your program's run succeeded by checking the job's status in the Job Manager.
To direct program output to a file, specify a file in PVF's Standard Output property before running the application. Program output can also be viewed directly in the Job Manager. To do so, start by selecting a job (upper pane) and task (lower pane) in the Job Manager. Use Action | Task Actions | View Task to open the Task Properties dialog. Program output will appear in the Output pane.
The debugging experience on a cluster is very similar to the local MPI debugging experience although you'll continue to use the Job Manager to track the status of jobs. Note that PVF cannot begin debugging a job that has been queued by the Job Manager; if this situation occurs it will appear that PVF has been suspended. To begin debugging, unqueue the job.
Once debugging starts, you should notice a difference in the processes window. In local MPI debugging, the only nodes used are local nodes. In cluster MPI debugging, remote nodes are also used.
If you find yourself switching back and forth between local and cluster MPI runs, you may want to create different project configurations using the Configuration Manager. For example, you might create a "Local" Configuration with all the properties set for working locally and a "Cluster" configuration for running on a cluster.
Good luck and don't forget to let us know how you're doing. Log on to the PGI User Forum or send questions to PGI Technical Support.