Re: Need help with large file reading and writing.

news.gatech.edu <liscon@xxxxxxxxx> wrote:

When I use fortran 90 code to read larger files,about 1G each, and
write the data to other opened files with 32-bit Intel-compiler. Each
time, I just opened 3 files simultaneously and closed unused ones. The
code always stops at the point of 10 files reading. I am not sure it is
the compiler problem about the limitation of buffer memory.

[sample code elided. I don't think it is telling me much about the
problem. I saw some "issues" in the sample code, but I'd guess they were
all artifacts of the attempt at simplification and have nothing to do
with the question. For example, the file names were... well, not what
one might think they were intended to be.]

Not a lot of data to go on, but I doubt it has anthing to do with
"buffer memory." Compilers don't need to buffer the whole file or
anything like that. Large records might cause buffer size "issues", but
I don't see any suggetsion that your record sizes are particularly
large.

My first guess would be that the compiler and/or operating system (you
didn't say what operating system that I noticed; the Intel compiler runs
on several) don't support files as large as you are trying to write.

My second guess would be a memory leak somewhere. If so, simplified
illustrative code would not be enough to find it. The old MS
Powerstation 4 compiler used to have memory leaks in formatted I/O. I
wouldn't expect that out of Intel, but I can't rule it out.

P.S. You mention files of gigabyte size, and your example shows
formatted I/O. I assume you know that formatted I/O is *VERY*
inefficient, both in terms of file size and time. Maybe you have good
reason to use formatted I/O anyway; maybe you don't even have a choice.
And maybe performance is not an issue. But I thought I'd mention it just
in case.

--
Richard Maine | Good judgment comes from experience;
email: my first.last at org.domain| experience comes from bad judgment.
org: nasa, domain: gov | -- Mark Twain
.