Re: Need help with large file reading and writing.


Thanks very much!
To clarify the problem:

Memory leak: I did use some allocatable matrice and forget to deallocate them, but I deallocate them or just declare them as static after you mentioned. The same problem happned. so maybe memory leak is not possible. The only parameters used often in the code are 7 integer variables.

Compilers: I use Intel Compiler(version 8.1) on Linux.

Also the code runs smoothly for 64 files when the files' size are small, say 100M each one. And when I increase the file size to about 400M each, it stops at the point of 32 files. Moreover, when the file size increases to about 1G, it just handles the first 10 files.

It always stops about 800 sec.

And the error output are:
Exited with exit code 1.

Resource usage summary:

CPU time : 832.00 sec.
Max Memory : 2 MB
Max Swap : 19 MB

Max Processes : 1
Max Threads : 1

The output (if any) follows:

Timeout alarm signaled

TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME
==== ========== ================ ======================= ===================
0001 compute-6- mpirun_ssh -np 1 Killed by PAM (SIGTERM) 07/13/2006 23:08:10


About the Formatted and Unformatted I/O, I don't know much about the efficiency. Thanks very much, I will try it right now.

Liscon


Richard E Maine wrote:

news.gatech.edu <liscon@xxxxxxxxx> wrote:


When I use fortran 90 code to read larger files,about 1G each, and write the data to other opened files with 32-bit Intel-compiler. Each
time, I just opened 3 files simultaneously and closed unused ones. The
code always stops at the point of 10 files reading. I am not sure it is
the compiler problem about the limitation of buffer memory.


[sample code elided. I don't think it is telling me much about the
problem. I saw some "issues" in the sample code, but I'd guess they were
all artifacts of the attempt at simplification and have nothing to do
with the question. For example, the file names were... well, not what
one might think they were intended to be.]

Not a lot of data to go on, but I doubt it has anthing to do with
"buffer memory." Compilers don't need to buffer the whole file or
anything like that. Large records might cause buffer size "issues", but
I don't see any suggetsion that your record sizes are particularly
large.

My first guess would be that the compiler and/or operating system (you
didn't say what operating system that I noticed; the Intel compiler runs
on several) don't support files as large as you are trying to write.

My second guess would be a memory leak somewhere. If so, simplified
illustrative code would not be enough to find it. The old MS
Powerstation 4 compiler used to have memory leaks in formatted I/O. I
wouldn't expect that out of Intel, but I can't rule it out.

P.S. You mention files of gigabyte size, and your example shows
formatted I/O. I assume you know that formatted I/O is *VERY*
inefficient, both in terms of file size and time. Maybe you have good
reason to use formatted I/O anyway; maybe you don't even have a choice.
And maybe performance is not an issue. But I thought I'd mention it just
in case.


.

Relevant Pages

  • Re: Memory leak using NewAnsiString
    ... > there are some memory leaks, most of them involving the use of strings, ... > The debugger says that 's' generates a memory leak, ... > freed automatically by the delphi compiler? ... Pass the 's' string as a var parameter to the function? ...
    (borland.public.delphi.language.objectpascal)
  • Re: Allocating memory for struct - when?
    ... > believe it is because there's some memory leak - the debugger tells me ... instead of "char". ... how would the compiler know how much memory to ... example of a situation that flexible array members were designed for. ...
    (comp.lang.c)
  • memory leak detection in evc
    ... Is there a way to detect memory leak in evc environment? ... as a test,i allocate a memory block and don't free it purposely, the compiler doesn't give any warning. ...
    (microsoft.public.pocketpc.developer)
  • Re: iso_varying_string in f2003?
    ... > possible to allocate something and then not deallocate it... ... That's not a memory leak. ... > to the array on return to the main program? ...
    (comp.lang.fortran)
  • Re: Not null feature with anonymous and named access types
    ... memory leak is almost certain. ... expose in the spec a not null type and in body use some nullable type for ... It is nice to have null excluded feature. ... Constraint_Error will not be raised when deallocate the access object ...
    (comp.lang.ada)