PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

CUDA-x86.

Compile Nemo Ocean with PGI + ACC to CUDA
Goto page 1, 2  Next
 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming
View previous topic :: View next topic  
Author Message
WRoX



Joined: 16 Nov 2012
Posts: 4

PostPosted: Tue Dec 18, 2012 1:34 pm    Post subject: Compile Nemo Ocean with PGI + ACC to CUDA Reply with quote

Hi,

We are a group of students from Technical University of Sofia, Bulgaria. We are trying to run the NEMO Ocean project on CUDA by using the PGI Accelerator placing directives in the .F90 fortran sources. We have nested "for" loops which we want to send to the CUDA cores for processing.

1. The directives we are using (a code snippet containing nested loops):

!$acc kernels loop
DO jk = 2, jpkm1
DO jj = 2, jpjm1
DO ji = fs_2, fs_jpim1 ! vector opt.
zwt(ji,jj,jk) = zwt(ji,jj,jk) + ah_wslp2(ji,jj,jk)
END DO
END DO
END DO


2. We have compiled the file containing the loops with command:
pgfortran -Mcuda -c file.F90.
The files which we include (USE in Fortran source) we compile to .mod with the following command: pgfortran -shared -fPIC include_file.F90

3. At this point we have the "file.o" + "file.mod".

4. After step 2 and 3 we compile NEMO Ocean with the following configuration file by USING gfortran compiler:

%NCDF_INC -I/usr/local/netcdf/include
%NCDF_LIB -L/usr/local/netcdf/lib -lnetcdf -lnetcdff
%FC gfortran
%FCFLAGS -I/usr/include/mpich2 -L/usr/lib/mpich2/lib -lmpich -lmpl -fdefault-real-8 -O3 -funroll-all-loops -fcray-pointer
%FFLAGS %FCFLAGS
%LD gfortran
%LDFLAGS -I/usr/include/mpich2 -L/usr/lib/mpich2/lib -lmpich -lmpl -L/usr/local/netcdf/lib -lnetcdf -lnetcdff
%FPPFLAGS -I/usr/include/mpich2 -L/usr/lib/mpich2/lib -lmpich -lmpl -P -C -traditional
%AR ar
%ARFLAGS -rs
%MK gmake
%USER_INC %NCDF_INC
%USER_LIB %NCDF_LIB

Note: The command we use to compile NEMO is:
./makenemo -m gfortran_linux_modify -r ORCA2_LIM -n MY_ORCA2_LIM -j 4

5. After the successful compilation of nemo a BUILD folder is generated. The folder contains all the ".o" and ".mod" files of NEMO.

6. Then we replace the successfuly compiled files which contain the nested for loops with the .o and .mod files produced by pgfortran (from step 3).

7. We comment the lines of NEMO Ocean's Make file so they are not being generated when we try to compile with the "modified" (step 3) files.

8. We compile assuring (step 7) that the new .o files will be used by pgfortran on new compilation. The following ERROR appears:

lib__fcm__nemo.a(trazdf.o): In function `__trazdf_MOD_tra_zdf':
trazdf.F90: (.text + 0x15f2): Undefined reference to `__trazdf_imp_MOD_tra_zdf_imp
trazdf.F90: (.text + 0x1818): Undefined reference to `__trazdf_imp_MOD_tra_zdf_imp
Collect2: error: ld returned 1 exist status
fcm_internal load failed (256)


- We would like to know if this is the right way (logically) to execute a separated fortran code (step 3) on the CUDA cores ?

- Is our approach correct i.e.:
Compile only the files containing Accelerator directives with pgfortran, or compile the entire NEMO Ocean project with pgfortran ?

Best Regards,
Technical University of Sofia,
Team of students

P.S. The university intends to buy a license for pgfortran if we manage to complete the project. We are using trial versions for now.
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 5815
Location: The Portland Group Inc.

PostPosted: Tue Dec 18, 2012 2:44 pm    Post subject: Reply with quote

Quote:
Compile only the files containing Accelerator directives with pgfortran, or compile the entire NEMO Ocean project with pgfortran ?
Since you're using F90 modules and presumably allocatable arrays, you'll need to use the same compiler for all files. Alternatively, you can write F77 code to bridge between the F90 codes, but this is often more trouble than it's worth.

Is there a reason my you can't compiler NEMO with PGI?
- Mat
Back to top
View user's profile
WRoX



Joined: 16 Nov 2012
Posts: 4

PostPosted: Wed Dec 19, 2012 7:57 am    Post subject: Reply with quote

Hi, Mat!
Thank you for your answer! We do not have a solid reason to compile a single file with pgfortran, and than the entire product using gfortran. We thought this was the correct way to go.
So we listened to your advice and compiled the entire product using pgfortran.
Before that we compiled mpich with pg compilers.
Config file for NEMO Ocean this time around is:


Code:
%NCDF_INC -I/home/kalin/installed/netcdf/include
%NCDF_LIB -L/home/kalin/installed/netcdf/lib -lnetcdf -lnetcdff
%FC pgfortran
%FCFLAGS -I/home/kalin/installed/mpich/include -L/home/kalin/installed/mpich/lib -lmpich -lmpl -shared -fPIC
%FFLAGS %FCFLAGS
%LD pgfortran
%LDFLAGS -I/home/kalin/installed/mpich/include -L/home/kalin/installed/mpich/lib -lmpich -lmpl -L/home/kalin/installed/netcdf/lib -lnetcdf -lnetcdff
%FPPFLAGS -I/home/kalin/installed/mpich/include -L/home/kalin/installed/mpich/lib -lmpich -lmpl -P -C -traditional
%AR ar
%ARFLAGS -rs
%MK gmake
%USER_INC %NCDF_INC
%USER_LIB %NCDF_LIB


This is the compile time error it gives me:

Code:
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 52)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_1d
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 67)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_2d
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 88)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_3d
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 105)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_2d_a
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 127)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_3d_a
fcm_internal compile failed (512)


Lines containing errors are as follows:
Code:
glob_sum_2d = SUM( ptab(:,:)*tmask_i(:,:) )
      IF( lk_mpp )   CALL mpp_sum( glob_sum_2d )


We declare these before hand:

Code:
 
INTERFACE glob_sum
      MODULE PROCEDURE glob_sum_2d, glob_sum_3d,glob_sum_2d_a, glob_sum_3d_a
END INTERFACE
   

   
   FUNCTION glob_sum_2d( ptab )
      REAL(wp), INTENT(in), DIMENSION(:,:) ::   ptab          ! input 2D array
      REAL(wp)                             ::   glob_sum_2d   ! global masked sum
      glob_sum_2d = SUM( ptab(:,:)*tmask_i(:,:) )
      IF( lk_mpp )   CALL mpp_sum( glob_sum_2d )
     
   END FUNCTION glob_sum_2d


As we were going through the forum it became apparent that there is a bug in the compiler itself, but as it turned out, it was affecting older versions.
Should we put some additional arguments to the compiler line, in order to make it go?
The thing is when we use gfortran everything goes well, not so with pgfortran (giving the error I mentioned).
We shouldn‘t be changing any NEMO Ocean Code, should we? Any ideas as of what our next step should be?
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 5815
Location: The Portland Group Inc.

PostPosted: Wed Dec 19, 2012 9:46 am    Post subject: Reply with quote

Quote:
PGF90-S-0155-Could not resolve generic procedure mpp_sum

This error indicates that the compiler can't resolve which mpp_sum to use. Can you post the interface block from mpp_sum? Does one of the routines take a "REAL(wp)" data type as an argument?

- Mat
Back to top
View user's profile
WRoX



Joined: 16 Nov 2012
Posts: 4

PostPosted: Wed Dec 19, 2012 11:55 am    Post subject: Reply with quote

Code:

INTERFACE mpp_sum
      MODULE PROCEDURE mpp_sum_a2s, mpp_sum_as, mpp_sum_ai, mpp_sum_s, mpp_sum_i
 END INTERFACE
   
   
   SUBROUTINE mpp_sum_as( parr, kdim, kcom )      ! Dummy routine
      REAL   , DIMENSION(:) :: parr
      INTEGER               :: kdim
      INTEGER, OPTIONAL     :: kcom
      WRITE(*,*) 'mpp_sum_as: You should not have seen this print! error?', kdim, parr(1), kcom
   END SUBROUTINE mpp_sum_as

   SUBROUTINE mpp_sum_a2s( parr, kdim, kcom )      ! Dummy routine
      REAL   , DIMENSION(:,:) :: parr
      INTEGER               :: kdim
      INTEGER, OPTIONAL     :: kcom
      WRITE(*,*) 'mpp_sum_a2s: You should not have seen this print! error?', kdim, parr(1,1), kcom
   END SUBROUTINE mpp_sum_a2s

   SUBROUTINE mpp_sum_ai( karr, kdim, kcom )      ! Dummy routine
      INTEGER, DIMENSION(:) :: karr
      INTEGER               :: kdim
      INTEGER, OPTIONAL     :: kcom
      WRITE(*,*) 'mpp_sum_ai: You should not have seen this print! error?', kdim, karr(1), kcom
   END SUBROUTINE mpp_sum_ai

   SUBROUTINE mpp_sum_s( psca, kcom )            ! Dummy routine
      REAL                  :: psca
      INTEGER, OPTIONAL     :: kcom
      WRITE(*,*) 'mpp_sum_s: You should not have seen this print! error?', psca, kcom
   END SUBROUTINE mpp_sum_s

   SUBROUTINE mpp_sum_i( kint, kcom )            ! Dummy routine
      integer               :: kint
      INTEGER, OPTIONAL     :: kcom
      WRITE(*,*) 'mpp_sum_i: You should not have seen this print! error?', kint, kcom
   END SUBROUTINE mpp_sum_i


We do not understand fortran so good, so what do you think about the snipped ?
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Accelerator Programming All times are GMT - 7 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group