PGI User Forum
 SearchSearch   MemberlistMemberlist     RegisterRegister   ProfileProfile    Log inLog in 

CUDA-x86.

PGFIO/stdio: File too large

 
Post new topic   Reply to topic    PGI User Forum Forum Index -> Programming and Compiling
View previous topic :: View next topic  
Author Message
kevindelorme



Joined: 03 Apr 2014
Posts: 1

PostPosted: Thu Apr 03, 2014 12:32 pm    Post subject: PGFIO/stdio: File too large Reply with quote

Hi,

I'm developing a python module that calls Fortran executable to convert RPN (Environment Canada) files to r2c. I do not have the wisdom of the ancients : I'm not familiar with Fortran.

I got this error on my logfiles :

PGFIO/stdio: File too large
PGFIO-F-/formatted write/unit=10/error code returned by host stdio - 27.
File name = /data/local/tmpdirs/afqscli/26968/tmpIw37tm/tmprOmNG-/TT_0-9950sg_2004-06.r2c formatted, sequential access record = 1
In source file fst2r2c.f90, at line number 302 (line 307 sometimes too)

Code:
301 !           Write time stamp to R2C file
302             write(iunr2c,'(A,I8,I8,A,A,A,A,A)'), &
303                ':Frame ',ip2,ip2,' "',strymd,' ',strhms,'"'
304
305 !           Write data to R2C file
306             do j = 1, yCount
307                write(iunr2c,trim(r2cFormat))(fldr2c(i,j),i=1,xCount)
308             enddo



This error occurs when the r2c file is larger than 2GB : the fortran executable seems to be unable to deal with big files. After searching for a while I found here a topic indicating I had to recompile the fst2r2c executable using the -Mlfs option. I actually did it but the error still occurs with files larger than 2GB.

Quote:
-Mlfs (32-bit Linux only) Link in the Large File Support routines
available on Linux versions later than Red Hat 7.0 or SuSE 7.1.
This will support files from Fortran I/O that are larger than
2GB. Equivalent to -L$PGI/linux86/7.1-4/liblf.


The program is run on this server :
Linux hawa 2.6.26-bpo.2-686-bigmem #1 SMP Fri Jul 3 21:38:05 UTC 2009 i686 GNU/Linux (Debian 32bits).

I wanted to know if the -Mlfs option (equivalent to -L$PGI/linux86/9.0/liblf in my case) was supported on my machine or not as I'm not on Fedora or SuSE. If not, should I consider moving my application to a 64bits server to fix that error ?

Regards,

kevindelorme
Back to top
View user's profile
cparrott



Joined: 02 May 2011
Posts: 146

PostPosted: Fri Apr 04, 2014 8:08 am    Post subject: Reply with quote

Hi Kevin,

I was able to construct the following program to successfully write out a > 2 GB file on a 32-bit system internally:

Code:

program iotest

  character :: a(1024,1024)
  integer :: i, j, f

  do i = 1, 1024
    do j = 1, 1024
      a(i,j) = 'x'
    end do
  end do

  open (f, file='myfile.txt', status='new')
  do i = 1, 2049
    write (f, *) a
  end do
  close (f)

end program iotest


I compiled it with PGI 9.0-5 as follows:

Code:

  pgf90 -o iotest -Mlfs iotest.f90


Running it on our 32-bit system here, I do get a large file:

Code:

cparrott /tmp $ ~/iotest
cparrott /tmp $ ls -l *.txt
-rw-rw-r--  1 cparrott sw 2203625736 Apr  4 07:56 myfile.txt
cparrott /tmp $


I just posted this to reassure both of us that this works, at least as of the PGI release you specified. :)

That said, there could be a number of reasons as to why your program is not working correctly. I notice you are running on a fairly old version of Debian - it is possible that there is an issue with that OS that is causing -Mlfs to not work correctly. We are not running that version of Debian here internally, and do not officially support it, so I cannot really reproduce the conditions you described here in-house. (However, we do support recent versions of Ubuntu, which is a fork of Debian.)

I do think moving to a 64-bit system would make this task a bit easier for you, though you might have to add the -mcmodel=medium flag if your application has a static data area that is larger than 2 GB. (This will just fail outright in 32-bit mode.) So, my initial suggestion would be to try moving over to a 64-bit system if you can. If your application still fails there, then let's see if we can work on reproducing your problem here internally.

Sorry for the long explanation - hope this helps.

Best regards,

+chris
Back to top
View user's profile
Display posts from previous:   
Post new topic   Reply to topic    PGI User Forum Forum Index -> Programming and Compiling All times are GMT - 7 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group