Re: Performance question

From: Paul Lalli (mritty_at_gmail.com)
Date: 03/29/05 

Date: Tue, 29 Mar 2005 18:35:47 GMT

Dave Sill wrote:
> One of my users made the following observation. I'm only an
> occasional, lightweight Perl user, so I can't explain what he's
> seeing. Can anyone shed some light on it? H/W is a pretty large/fast
> Dell server running RHEL 3.
>
> ----
> I manufactured a 401x401 [ linearly =160801 element] array [@judy]
> each element having string values like
>
> 01000000001110000000000000000001
>
> I needed to make a comma delimited ascii file of this data.
>
> I decided a single IO write of a string would be the fastest, so i
> made a string
> $str="";
> foreach $i(0..$#judy-1)
> {
> $str=$str."$judy[$i],"
> }
> $str=$str."$judy[$#judy]"; open(OUT,">$output_file");print OUT
> $str;close(OUT);
> `gzip -f $output_file`;
>
> this took 16 minutes.
>
> i tried it the slow way,
>
> open(OUT,">$output_file");
> foreach $i(0..$#judy-1)
> {
> print OUT "$judy[$i],";
> }
> print OUT "$judy[$#judy]"; close(OUT);
> `gzip -f $output_file`;
>
> with 160K IOs, this took about 3 seconds.
>
> the gz files were different, but diff said uncompressed they were the
> same.

Your user has an odd definition of "faster" and "slower". I don't know
what would make the user think that storing the entire 160,801 element
array in memory TWICE would be faster than just printing what's needed
when it's needed.

In the first algorithm, the user is storing one large string, and each
time through the loop, appending to that string. Towards the end, this
means storing over (160,000 x 32) bytes in a single scalar, and asking
perl to append to the end of that string. Then finally you ask perl to
make one absurdly large I/O access.

In the second algorithm, you're simply printing 32 bytes repeatedly.

Neither of those ways are especially good perl code, of course. The
first would be better written:

my $str = join (',', @judy);
open my $out, '>', $output_file or die "Can't open output: $!";
print $out $str;
close $out;

The second would be better written

open my $out, '>', $output_file or die "Can't open output: $!";
{
    local $, = ',';
    print $out @judy;
}
close $out;

I would suggest your user use the Benchmark module to determine which of
these is actually faster.

Paul Lalli

Relevant Pages

  • Cant Compiling perl5.8.8 on FreeBSD6.2
    ... WITH_DEBUGGING=yes Build perl with debugging support. ... First let's make sure your kit is complete. ... What is the file extension used for shared libraries? ... I'll use sprintf to convert floats into a string. ...
    (comp.unix.bsd.freebsd.misc)
  • Re: use of DBI; I am getting multiple error messages mixed in with ?the correct output.
    ... But I'm not talking about C++ or Java or SQL. ... This is a perl newsgroup. ... to means by a "defined null string", as opposed to an undefined value. ... And where exactly are you getting the idea that the empty string is a ...
    (comp.lang.perl.misc)
  • Re: Perl Strings vs FileHandle
    ... Just wanted to run this through Perl gurus to see if fit is ... testing it every time through the loop. ... The comparison test to the string '1' is superfluous. ... Shared Hosting, Reseller Hosting, Dedicated & Semi-Dedicated servers ...
    (comp.lang.perl.misc)
  • Re: Working with Source Code to Insert Copyright Statements as a Header
    ... Paul Lalli wrote: ... Where could I go to find some samples of PERL code that do the ... ascii english text, etc etc etc) at the top of the file, from a string: ... `perldoc Tie::File` at the command line to read its documentation. ...
    (comp.lang.perl.misc)
  • Re: Perl ActiveX: A VBA string passed to my control is treated as a doubel-quoted string
    ... I have written a Perl ActiveX control that's suppose to do some regular ... When I use this string in VBA, I get erroneous resutls from my control. ... Perl regex has its own set of escape characters like any other parser. ...
    (comp.lang.perl.misc)