Re: two or more programs writing a single file



On Aug 2, 7:38 pm, nos...@xxxxxxxxxxxxx (Richard Maine) wrote:
Gabor <g...@xxxxxxxx> wrote:
Can it be some subtle gfortran bug?

Unlikely. Having multiple programs write to the same file without
locking is a bug (in your application). It is a bug that can produce
exactly the symptoms you describe. Since you have a known bug in your
application that matches the symptoms, but no information pointing to
anything about the compiler, I see no basis for such a suspicion. It
seems rather "random". While compilers do have bugs, it sometimes seems
like those who are most prone to suspect the compiler are those least
likely to have actually found one of the compiler bugs. As a general
rule, one should suspect one's own code first. That's not always the
source fo the problem, but it is still a good general rule to start
there.

In your case, you have indeed found a problem there, which makes it even
less fruitful to go on to suspect the compiler anyway.

I am somewhat disappointed because these "long" calculations
often take only tens of minutes, and the bookkeeping
can take away most of the gain of quick answers.

I'm a little surpised that the work to combine the files would take a
significant fraction of the tens of minutes that you describe unless
there is an awfully lot of output. If there is really that much output,
your application might be inherently output limitted. That isn't
horribly unusual. You say that the app is "embassassingly parallel." In
that case may well be that serialization of the output is the limitting
factor. The ration of I/O to computation is a fundamental factor in
extreme parallelization.

Do note that if I/O performance is being a hindrance, you might want to
consider using unformatted I/O instead of formatted for the intermediate
files. Formatted I/O is far slower than unformatted, often by a factor
of 10 or so. Even if the final file needs to be formatted for human
consumption or other purposes, the intermediate files could be
unformatted.

I suppose that another possible inefficiency could be in the OPENs.
Opening a file typically is relatively slow, compared to such things as
writing a single record to it. If, perhaps, each of your files is very
small (for example, if they are only one line each), it might be that
you are spending a lot more time on the open/close than on
reading/writing the data. If that's the case, possibly you could benefit
from more complicated schemes of interprocess communication... but they
would be quite a bit more complicated (and I couldn't tell you off the
top of my head how to do them; it would take some research).

--
Richard Maine                    | Good judgement comes from experience;
email: last name at domain . net | experience comes from bad judgement.
domain: summertriangle           |  -- Mark Twain

Thank you for the detailed explanation.
I learn my lesson and do understand
that I have to find a solution in my code.
Sorry, for any hint of a gfortran bug,
it is great that it is there and we can use it.
.