Re: speed up calculation suggestions

<rleavitt@xxxxxxxxxxxxxxxx> wrote:

I process many millions of records from text files through some fairly
simple calculations, and write out the results. I want to speed up this
process

I just seconds ago finished posting to another thread of someone writing
very general questions about optimization. It seems to be the subject of
the day. While the general advice there (look into improvement of
algorithms before code tweaks) is almost always applicable, you did
mention a term that sets off knee-jerk reactions.

That term is "text files". Text file I/O is notoriously slow - very
slow. My personal rough rule of thumb is that it is often about a factor
of 10 slower than unformatted I/O. That is just a rule of thumb and is
not precise at all, but it should serve to illustrate the "very slow". A
lot of this is from the conversion between text and internal numeric
forms.

If your calculations are "fairly simple", then the odds are high that
the run-time of your program could be dominated by the text file I/O.

Do the files *HAVE* to be text? Getting away from text is likely to save
a lot. If the same files are rocessed multiple times, it coudl well be
worthwhile to add a preprocessing step to first convert the text file
into an unformatted file; that way you only have to read the text file
once. Likewise, even if the input and final output files have to be text
for compatibility with other uses, one could make any intermediate files
unformatted.

There can sometimes be ways to somewhat speed up text file I/O, but that
gets into code tweaks (and is also likely to be at least somewhat
compiler dependent).

--
Richard Maine | Good judgement comes from experience;
email: last name at domain . net | experience comes from bad judgement.
domain: summertriangle | -- Mark Twain
.

Relevant Pages

  • Rule of Thumb for coax chokes
    ... an original rule of thumb. ... a coax choke, when deciding how many turns of coax to put ... Backup calculations will be published on my web page. ...
    (rec.radio.amateur.antenna)
  • Re: Line per inch
    ... In article, Aandi Inston ... And because the rule of thumb works in the real world, ... calculations are unneccesary at the user level. ... fish. ...
    (comp.publish.prepress)