PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

Free OpenACC Course

Setting MPI Processor/Core Affinity

 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Programming and Compiling
View previous topic :: View next topic  
Author Message
TheMatt



Joined: 06 Jul 2009
Posts: 340
Location: Greenbelt, MD

PostPosted: Tue Jul 13, 2010 9:39 am    Post subject: Setting MPI Processor/Core Affinity Reply with quote

Is there a way to set core/processor affinity with PGI's default MPICH setup?

That is, I have a dual-Nehalem system so two, quad-core chips. If I request '-np 4' I'd like to know if I'm getting 4 cores on one processor or 2 cores on 2 processors and how I can select that.

Thanks,
Matt
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6741
Location: The Portland Group Inc.

PostPosted: Tue Jul 13, 2010 10:20 am    Post subject: Reply with quote

Hi Matt,

The order is determined by the order in which the machines appear in the machines.LINUX file.

This would round-robin:
Code:
node01
node02
node01
node02
node01
node02
node01
node02


This would set the first two on node01, and the second two on node2
Code:

node01:2
node02:2
node01:2
node02:2


This would set the first four on node01, and the next four on node02:
Code:
node01:4
node02:4



- Mat
Back to top
View user's profile
TheMatt



Joined: 06 Jul 2009
Posts: 340
Location: Greenbelt, MD

PostPosted: Tue Jul 13, 2010 10:25 am    Post subject: Reply with quote

mkcolg wrote:
The order is determined by the order in which the machines appear in the machines.LINUX file.

Okay. Couple of questions. First, let's say my machine's short hostname is "teslabox". My /opt/pgi/linux86-64/2010/mpi/mpich/share/machines.LINUX file is:
Code:
teslabox

Just one word. What does this mean? Round-robin?

And, is there anyway to dynamically change the order? I'm pretty sure Open MPI and MVAPICH have ways to set this via an environment variable or commandline switch?
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6741
Location: The Portland Group Inc.

PostPosted: Tue Jul 13, 2010 11:11 am    Post subject: Reply with quote

Hi Matt,

Sorry, I mis-read your OP. You have a single 2-socket system, not 2 1 socket systems.

Setting single system core affinity is a bit more difficult since this is done by the OS not mpirun. What I have done in the past is write a wrapper script which sets the affinity (via taskset or numactl) then run the application. mpirun then runs the wrapper instead of the application.

The problem is knowing the rank of each individual process. In MPICH2, mpiexec sets the environment variable "PMI_RANK" which can be used, but I don't think MPICH has an equivalent.

Here's a simple Perl example. Note that PMI_SIZE is set to the total number of MPI processes and can be used to create more complex affinity bindings.
Code:
% cat penv.pl
#!/usr/local/bin/perl -w

use strict;

my $cmd = "numactl --physcpubind=$ENV{PMI_RANK} ";
my $arg = "";
while ($arg = shift) {
  $cmd .= $arg . " ";
}
exec($cmd);

% mpiexec -np 4 penv.pl date
Tue Jul 13 11:07:10 PDT 2010
Tue Jul 13 11:07:10 PDT 2010
Tue Jul 13 11:07:10 PDT 2010
Tue Jul 13 11:07:10 PDT 2010



Quote:
I'm pretty sure Open MPI and MVAPICH have ways to set this via an environment variable or commandline switch?
I think so as well, but I'm not sure of the details.

- Mat
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Programming and Compiling All times are GMT - 7 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group