PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

CUDA-x86.

nvidia-smi and exclusive compute mode

 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming
View previous topic :: View next topic  
Author Message
xray



Joined: 21 Jan 2010
Posts: 85

PostPosted: Tue Apr 20, 2010 2:15 am    Post subject: nvidia-smi and exclusive compute mode Reply with quote

Hi,
we have a Tesla S1070 system and using 2 of its GPUs in an Linux environment. Now we want to use the GPUs in batch mode. I read that using the Nvidia driver tool nvidia-smi is quite useful for scheduling the jobs to the both GPUs because you can set the GPUs to exclusive compute mode and thus it is guranteed that only one job can run on one (free) GPU. I tried it with CUDA C and everything is working fine.
However, I want to work it with PGI Acc C/Fortran (and CUDA Fortran) as well. Is there any support for this feature?
I tried a small test application with PGI Acc C (and just starting a couple of these Acc programs which do not set explicitly a device). I got "call to cuCtxCreate returned error 999: Unknown". ACC_NOTIFY shows that there is only one kernel launched (device 0).
So, do you know how I could force the program to let it scheduled itself by the nvidia driver tool?
Regards, Sandra
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6072
Location: The Portland Group Inc.

PostPosted: Fri Apr 23, 2010 1:10 pm    Post subject: Reply with quote

Hi Sandra,

I'm not familiar with nvidia-smi myself, but CUDA Fortran is interoperable with CUDA C. Can you call a CUDA C routine to perform the task?

In general, GPU scheduling needs to be performed by the user.

- Mat
Back to top
View user's profile
xray



Joined: 21 Jan 2010
Posts: 85

PostPosted: Sun Apr 25, 2010 11:57 pm    Post subject: Reply with quote

Hello Mat,
with CUDA C and CUDA Fortran everything is working fine (see source code below). That means, if I don't use cudaSetDevice in my program, the kernel is executed on a available GPU (if there is one)... and that is how it should work. (Thus, if the user does not request a certain GPU, the automatic scheduling works well).
However, using PGI Accelerator programming model my program always wants to run on device 0. That's the problem.
But PGI Accelerator is based on CUDA as well, isn't it? Thus, I assume there must be somewhere an explicit call to set the cuda device to 0 in the PGI Accelerator implementation. If that is the point, is there a reason for setting the device explicitly? Otherwise I would appreciate it, if it could somehow removed. I think, this might get a general problem for all batch mode usage of GPUs that wants to use PGI Accelerator.


This is a small cuda fortran source code which is executed 3 times (we have 2 GPUs): 2 times it is scheduled to GPU 0/1 and one time I am getting an error:
Code:
program MAIN
        use cudafor
        implicit none
        integer, device, allocatable:: d_ptr(:)
        integer:: error, dev
        allocate(d_ptr(1))
        error = cudaGetLastError()
        if (error /= cudaSuccess) then
                write(*,*) 'Error: ', cudaGetErrorString(error)
                stop
        end if
        error = cudaGetDevice(dev)
        write(*,*) "Running on device ", dev
        call sleep(10)
        deallocate(d_ptr)
end program MAIN

And this is the shell script I'am using:
Code:
pgfortran -Mcuda=3.0 devtest.CUF -o devtest;
for i in 1 2 3; do ./devtest & done;

The output looks like the following (order of output statements differs):
Code:
0: ALLOCATE: 4 bytes requested; status = 38
 Running on device             1
 Running on device             0



BTW: If you want to try it out, you have to set your GPUs to exclusive compute mode first. You can do that by: "nvidia-smi -g 0 -c 1" where -g denotes the ID of the GPU (so you probably have to do "nvidia-smi -g 1 -c 1", too) and -c specifies the compute mode: 1 means exclusive compute mode (0 is the default). Test it by using: "nvidia-smi -s" -> All GPUs and their compute mode number are listed.
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6072
Location: The Portland Group Inc.

PostPosted: Mon Apr 26, 2010 12:50 pm    Post subject: Reply with quote

Hi Sandra,

In the PGI Accelerator model the default is to use device 0. However you can set which device to use by calling "acc_set_device_num" from your program or use the environment variable ACC_DEVICE_NUM. (See page 23 and 27 of http://www.pgroup.com/lit/whitepapers/pgi_accel_prog_model_1.2.pdf)

Does this solve the problem?

- Mat
Back to top
View user's profile
xray



Joined: 21 Jan 2010
Posts: 85

PostPosted: Tue Apr 27, 2010 3:46 am    Post subject: Reply with quote

Now, that's unfortunately not helping. Because for setting the device explicitly (with acc_set_device_num or the environment variable) you already have to know which GPU is available and which is not. But as a normal user in a cluster multi-user environment you don't know that.
So I'm afraid we have to implement it somehow on our own with SGE. I just thought there might be a software/driver solution for the PGI Accelerator model, as there is for CUDA.
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming All times are GMT - 7 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group