perl hash speed and memory efficiency

From: odigity (ofer_at_netapt.com)
Date: 05/27/04 

Date: 27 May 2004 10:50:18 -0700

I'm running some tests to try to gauge the speed of perl's hashing
abilities, as well as the memory footprint. I wrote this function:

sub buildhash
{
  my %hash;
  foreach my $foo (1..100_000) {
    foreach my $bar ('a'..'z') {
      $hash{"$foo.$bar"} = 1;
    }
  }
  undef %hash;
}

Then I threw in the Benchmark module, like this:

timethis( 1, "buildhash()" );

It seems to use about 200MB of memory for 2.6 million small key/value
pairs, which is pretty efficient (~80bytes/pair). However, it doesn't
release the memory after the undef (I checked by stopping execution at
that point with a sleep statement and studying memory usage with
`ps`).

It either takes 11 seconds or 75 seconds depending on how I execute
it. Let me explain.

I first tried running it once, and it took 11 seconds. I tried twice,
and it took 86. This didn't make any sense to me. Here's the command
I used:

timethis( 2, "buildhash()" );

Then I tried unrolling it like this:

timethis( 1, "buildhash()" );
timethis( 1, "buildhash()" );

And that took 22 seconds (11/each), as I expected the first time.

So the question that is most driving me crazy is: For the sake of
Pete, why the difference!?

-ofer

Relevant Pages

  • Re: [Full-disclosure] [Dailydave] What RedHat doesnt want you to know about ExecShield (without
    ... buffer overflow attacks by performing executable memory checks. ... This is not the case with ExecShield without NX. ... code execution, in the other you do not. ...
    (Full-Disclosure)
  • [NT] Defeating Microsoft Windows XP SP2 Heap Protection and DEP Bypass
    ... The following security advisory is sent to the securiteam mailing list, and can be found at the SecuriTeam web site: http://www.securiteam.com ... and bypassing DEP (Data Execution Prevention). ... Buffer overrun attacks are among the most common mechanisms, or vectors, ... a long string to an input stream or control longer than the memory ...
    (Securiteam)
  • Re: [SLE] Threaded Perl
    ... > Pentium-III Hardware because if the problem is essentially memory bound ... I'm not familiar with the breakdown of execution units and how they relate ... that patterns of primary storage access (especially overall L2 and L3 ... execution patterns, access to RAM is the limiting factor. ...
    (SuSE)
  • Re: Can you write code directly in CIL ???
    ... I don't care if a GC occurs during the execution of my code. ... always needs all of this memory the whole time that it is executing. ... and the CLR isn't going to care what your function is doing. ... >>> You don't understand a fundamental concept to .NET and CIL. ...
    (microsoft.public.dotnet.languages.csharp)
  • RE: Defeating Microsoft Windows XP SP2 Heap protection
    ... Fact is XP SP2 is still far less likely to be vulnerable to buffer overflow ... > than the memory buffer allocated to hold it. ... > the operating system is now more careful to reduce both stack and heap ... > Execution Protection ...
    (microsoft.public.windowsxp.general)