Re: two or more programs writing a single file



On Aug 2, 7:38 pm, nos...@xxxxxxxxxxxxx (Richard Maine) wrote:
Gabor <g...@xxxxxxxx> wrote:
Can it be some subtle gfortran bug?

Unlikely. Having multiple programs write to the same file without
locking is a bug (in your application). It is a bug that can produce
exactly the symptoms you describe. Since you have a known bug in your
application that matches the symptoms, but no information pointing to
anything about the compiler, I see no basis for such a suspicion. It
seems rather "random". While compilers do have bugs, it sometimes seems
like those who are most prone to suspect the compiler are those least
likely to have actually found one of the compiler bugs. As a general
rule, one should suspect one's own code first. That's not always the
source fo the problem, but it is still a good general rule to start
there.

In your case, you have indeed found a problem there, which makes it even
less fruitful to go on to suspect the compiler anyway.

I am somewhat disappointed because these "long" calculations
often take only tens of minutes, and the bookkeeping
can take away most of the gain of quick answers.

I'm a little surpised that the work to combine the files would take a
significant fraction of the tens of minutes that you describe unless
there is an awfully lot of output. If there is really that much output,
your application might be inherently output limitted. That isn't
horribly unusual. You say that the app is "embassassingly parallel." In
that case may well be that serialization of the output is the limitting
factor. The ration of I/O to computation is a fundamental factor in
extreme parallelization.

Do note that if I/O performance is being a hindrance, you might want to
consider using unformatted I/O instead of formatted for the intermediate
files. Formatted I/O is far slower than unformatted, often by a factor
of 10 or so. Even if the final file needs to be formatted for human
consumption or other purposes, the intermediate files could be
unformatted.

I suppose that another possible inefficiency could be in the OPENs.
Opening a file typically is relatively slow, compared to such things as
writing a single record to it. If, perhaps, each of your files is very
small (for example, if they are only one line each), it might be that
you are spending a lot more time on the open/close than on
reading/writing the data. If that's the case, possibly you could benefit
from more complicated schemes of interprocess communication... but they
would be quite a bit more complicated (and I couldn't tell you off the
top of my head how to do them; it would take some research).

--
Richard Maine                    | Good judgement comes from experience;
email: last name at domain . net | experience comes from bad judgement.
domain: summertriangle           |  -- Mark Twain

Thank you for the detailed explanation.
I learn my lesson and do understand
that I have to find a solution in my code.
Sorry, for any hint of a gfortran bug,
it is great that it is there and we can use it.
.



Relevant Pages

  • Re: Text box throws an Error 7 Out of Memory with less than 27K in it!
    ... I'd have to agree with Bob here, ... >> those exact numbers and then concatenates that many more, ... I used to work supercomputer compiler support at CDC. ... > My favorite such bug at CDC was a benchmark that ran instantaneously. ...
    (microsoft.public.vb.enterprise)
  • Re: Bug in latest IAR MSP430 compiler optimization???
    ... It has been assigned bug id EW20095, and it will be fixed in the ... The tests compiler companies do are quite extensive. ... register allocation when combined with post-incs and rollup. ... IAR use an industry standard test suite like this (that you don't ...
    (comp.arch.embedded)
  • Re: Salford compiler: Assigning a value to a constant expression is invalid
    ... a bug, it is not usually wise, I think, to assume that it's a compiler bug. ... -|> I'd suggest submitting a self-contained source code file to Salford Support. ... So all I had to do in that case was send the routine and ...
    (comp.lang.fortran)
  • Re: Another computer algebra system : smib
    ... Certainly sometimes a bug is discovered only after a program is ported to a new environment -- a bug that existed in the old environment too, but just had not been exercised. ... Spending time porting a program is ... If the code is a command line drive program, then there should be few differences between for platform A and code for platform B. If one uses portable thread libraries, then if code breaks on platform B, but not on A, I would certainly think it is worth while spending time on platform B to find the problem, as it is highly likely to be a logical error in the program. ... The fact this is new code, I would not release it with over 1600 warnings messages from the compiler. ...
    (sci.math.symbolic)
  • Re: timeouts
    ... If I report a bug in the GCC C compiler, for example, then I can ... Stephen> a support contract for it. ... option when exclusively developing open source software. ...
    (comp.lang.ada)