PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

CUDA-x86.

Large file support on MPI?

 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Programming and Compiling
View previous topic :: View next topic  
Author Message
danish



Joined: 29 Mar 2005
Posts: 6

PostPosted: Tue Mar 29, 2005 2:17 am    Post subject: Large file support on MPI? Reply with quote

Hi,

Currently we have using PGI5.1-3 Server version with MPICH 1.2.6 on Athlon XP system.
Everything works fine except large file support on mpi.
I have choose -Mlfs during compiling the codes.
However, the executing always stop and error happen when output file larger than 2GB.
If only with pgf90(or 77) -Mlfs, it works fine.
Is this trouble causing by installing MPICH without -Mlfs configuration?
Currently MPICH are installed with your suggestion option (list in FAQ).
Any suggestions?
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6120
Location: The Portland Group Inc.

PostPosted: Tue Mar 29, 2005 9:42 am    Post subject: Reply with quote

Hi Danish,

Although I have not tried this myself, I believe you need to reconfigure and rebuild your MPICH library using the following define flags in order to support large files. Configure your installation as normal, but instead of setting CC to pgcc set it to:
Code:
setenv CC "pgcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE"

Everything I've read says that this should do it, but please let us know how it does.

Thanks,
Mat
Back to top
View user's profile
danish



Joined: 29 Mar 2005
Posts: 6

PostPosted: Wed Mar 30, 2005 3:14 am    Post subject: Reply with quote

Thanks for your reply!

According your suggestion, I change the configuration option and rebuild again, the command is like below:

-----------------------------------------------------------------
env CC=pgcc CFLAGS='-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Msignextend -tp px' FC=pgf77 FFLAGS='-tp px' CXX=pgCC CCFLAGS='-tp px' F90=pgf90 F90FLAGS='-tp px' MAKE=gmake ./configure --enable-debug -opt=-fast -prefix=/usr/local/mpich
gmake -f Makefile
gmake install
-----------------------------------------------------------------

However, the problem still happen when the file size exceed 2GB.
The following is error message:

------------------------------------------------------------------------------------
p1_28959: p4_error: net_recv read: probable EOF on socket: 1
p3_2338: p4_error: net_recv read: probable EOF on socket: 1
p2_4167: p4_error: net_recv read: probable EOF on socket: 1
/usr/local/mpich/bin/mpirun: line 1: 15803 File size limit exceeded(core dumped) /home/danish/run/ECS/shelf -p4pg /home/danish/run/ECS/PI15677 -p4wd /home/danish/run/ECS
Broken pipe
Broken pipe
Broken pipe
-----------------------------------------------------------------------------------

I also use -Mlfs during compilation, like
mpif90 -Mlfs -tp athlonxp -r8 -O2 ***.f90

Still not work. :(
Any ideas?
Back to top
View user's profile
mkcolg



Joined: 30 Jun 2004
Posts: 6120
Location: The Portland Group Inc.

PostPosted: Wed Mar 30, 2005 9:52 am    Post subject: Reply with quote

Hi Danish,

I did a bit more digging and per the ROMIO user's guide in section 4.1 the correct define flags are "-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE" not "-D_LARGEFILE_SOURCE" as I saw posted on the web. These are glibc definitions included in the 2.4 and newer kernels, so make sure your kernal supports this.

Try again using the new define flags. Although pgcc/pgCC should work, if it doesn't try using gcc/g++ instead. There may be some GNUisms I'm unaware of. If it still doesn't work, try contacting the ROMIO team (See: http://www-unix.mcs.anl.gov/romio) . Let us know how it goes!

Thanks,
Mat
Back to top
View user's profile
danish



Joined: 29 Mar 2005
Posts: 6

PostPosted: Thu Mar 31, 2005 2:39 am    Post subject: Reply with quote

Hi mkcolg,

I have try the new flags with PGI and read ROMIO user guide.
Seems still not working with this.
However, I found there is a large file test in ROMIO directory.
This test is not working on my system.

There are some questions came out after try this and read the manual.
Althrough ROMIO is combine in MPICH(1.2.6), I am not very sure ROMIO was installed properly with MPICH standard install procedure.

Second, ROMIO seems only work with MPICH2.
I am not very sure about this.
If so, MPICH2 need to be installed.
I found MPI-2-C++ diretory in mpich install directory, also ROMIO.
Is this means these will be installed properly?
I have check the ROMIO directory and found "romioinstal"l script in "util".
The default install path is /usr/local/romio.
If ROMIO is install properly, seems /usr/local/romio should be there.
But I can't find this directory in /usr/local, even in /usr/local/mpich(my mpich install prefix).
So I wondering ROMIO and MPICH2 are not installed, this cause the CFLAGS won't take any effects.

Sorry to ask you some questions not directly relating to PGI.
And thanks for your help.
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Programming and Compiling All times are GMT - 7 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group