PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

CUDA-x86.

Unknown 8GB memory getting allocated on GPU
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming
View previous topic :: View next topic  
Author Message
anirbanjana



Joined: 11 Aug 2012
Posts: 28

PostPosted: Wed Sep 25, 2013 5:46 pm    Post subject: Reply with quote

Hi Mat,
These variables are all 3X1 arrays of type double, except the loop index LL which is a scalar. The list contains 20 arrays, and with 18486 copies of each, the total space needed should be a mere 8.5 MB (=20*8*3*18486/1024/1024). There are some more large shared arrays, but I have estimated that the storage requirement should be of the order of tens of MBs, nowhere close to 8GB!

In fact, I have reverse engineered where the big number 8194917744 (~8GB) is coming from. 8194917744=18486*18471*24 !!!!! So a copy each of a 3x1 double array is being attempted for a grid of 18486*18471 threads. But since my inner loops are much smaller (as well as the fact that I have made them sequential), I can't see what this large grid of threads is associated with. May be I am missing sth (it is a complex piece of code). Is there a way to generate more illuminating diagnostic messages? Can I get the code to print an error message that says exactly what variable or loop the large memory allocation is associated with?

Thanks very much
Anirban
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6120
Location: The Portland Group Inc.

PostPosted: Thu Sep 26, 2013 8:19 am    Post subject: Reply with quote

Quote:
May be I am missing sth (it is a complex piece of code). Is there a way to generate more illuminating diagnostic messages?
You can set the environment variable "PGI_ACC_DEBUG=1".

- Mat
Back to top
View user's profile
anirbanjana



Joined: 11 Aug 2012
Posts: 28

PostPosted: Thu Sep 26, 2013 10:09 am    Post subject: Reply with quote

Hi Mat,
I already did that. Excerpt of the output of PGI_ACC_DEBUG as well as from cuda-gdb is in my original post that started this thread. The issue seems to occur right after the shared array "neighbor" was copied from host to device, but I am not sure if "neighbor" is the issue or sth else.

Can you see additional clues in the posted output?

Thanks a lot
Anirban
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6120
Location: The Portland Group Inc.

PostPosted: Thu Sep 26, 2013 2:35 pm    Post subject: Reply with quote

Hi Anirban,

Are you able to share the code? If I can reproduce the issue, it will give me a much better understanding of the problem.

Though, some other things for you try is to explicitly set the outer loop schedule to "gang, vector", and possibly use fixed widths (num_gangs, vector_length) to reduce the number of threads. It doesn't explain where the 8GB is coming from, but if the 8GB reduces, then at least we know it's one of the private arrays.

Also, try compiling with "-acc=noautopar" so the compiler doesn't auto vectorize the inner loops. My thought here is that

- Mat
Back to top
View user's profile
anirbanjana



Joined: 11 Aug 2012
Posts: 28

PostPosted: Fri Sep 27, 2013 10:47 am    Post subject: Reply with quote

Hi Mat,
Sure, let me know how to share it. The code is MFIX, which you mentioned earlier that you have access to. So I can send you just the file calc_force_des.f I am modifying, as well as the input/restart data.

I am trying out the fixed gang/worker/vector configuration. I do think it is one of the private arrays (probably des_pos_new, as per the cuda-gdb output in the first post?). But why it is copied 18Kx18K times is the mystery I would like to resolve.

Thanks
Anirban
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming All times are GMT - 7 Hours
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group