PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

Free OpenACC Webinar

Issue when using multiple GPUs

 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming
View previous topic :: View next topic  
Author Message
KarlWilkinson85254



Joined: 17 Jan 2013
Posts: 8

PostPosted: Tue Apr 22, 2014 6:09 am    Post subject: Issue when using multiple GPUs Reply with quote

Hi,

I am seeing the following error. There seems to be something wrong outside my code as I am using "-ta=tesla:cc35" (which seems to be reflected in the message itself). Also, the code works on other machines in this context.

Cheers,

Karl

Code:

mpirun -np 4 ../../bin/nvidia.tesla_cc35_cuda55 > test
The accelerator does not match the profile for which this program was compiled
Current file:     /home-2/kwilkinson/ONETEP_3.5.9.11/devel/src/kinetic_mod.F90
Current function: kinetic_gpu_app2_func_batch
Current line:     669
Current region was compiled for:
NVIDIA Tesla GPU sm30 sm35
Available accelerators:
device[1]: NVIDIA Tesla GPU 1, compute capability 3.5
device[2]: NVIDIA Tesla GPU 2, compute capability 3.5
device[3]: NVIDIA Tesla GPU 3, compute capability 3.5
device[4]: NVIDIA Tesla GPU 4, compute capability 3.5
device[5]: NVIDIA Tesla GPU 5, compute capability 3.5
device[6]: NVIDIA Tesla GPU 6, compute capability 3.5
device[7]: Native X86 (CURRENT DEVICE)
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6218
Location: The Portland Group Inc.

PostPosted: Tue Apr 22, 2014 11:13 am    Post subject: Reply with quote

Hi Karl,

How are you assigning MPI processes to GPUs? It looks like the code it trying to run on device 7, which is the CPU.

Quote:
device[7]: Native X86 (CURRENT DEVICE)


- Mat
Back to top
View user's profile
KarlWilkinson85254



Joined: 17 Jan 2013
Posts: 8

PostPosted: Tue Apr 22, 2014 12:05 pm    Post subject: Reply with quote

Hi Mat,

That was my first thought, and I tried to use export CUDA_VISIBLE_DEVICES=0,1,2,3 accordingly, with no joy. I am using acc_set_device_num within the code to map GPUs to MPI ranks.

I also tried using 3 MPI ranks with either 0,1,2 or 1,2,3 visible and saw the same thing in both cases, albeit with one less GPU in the error message list.

BTW, this is on the PSG cluster.

Cheers,

Karl
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6218
Location: The Portland Group Inc.

PostPosted: Wed Apr 23, 2014 9:45 am    Post subject: Reply with quote

Hi Karl,

I saw your notes to Adam and it appears that you determined that this was an issue in how you were calling acc_set_device_num.

- Mat
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming All times are GMT - 7 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group