Re: perl multithreading performance

On Wed, 27 Aug 2008 12:59:36 -0700, dniq00 wrote:

Everything works fine, HOWEVER! It's all so damn slow! It's only 10%
faster than single-process script, consumes about 2-3 times more memory
and about as much times more CPU.

I've also tried abandoning the Thread:Queue and just use threads::shared
with lock/cond_wait/cond_signal combination, without much success.

I've tried to play with $cpu_count and $buf_size, and found that after
$buf_size > 1000 doesn't make much difference, and $cpu_count > 2
actually makes things a lot worse.

Any ideas why in the world it's so slow? I did some research and
couldn't find a lot of info, other than the way I do it pretty much the
way it should be done, unless I'm missing something...

Hope anybody can enlighten me...

THANKS!

The speed of perl's threading is dependent on how much you share between
threads. Sharing the lines before processing them can become a
bottleneck, I suspect that's the problem in your case. You probably want
to divide the work first, and only used shared resources to report back
the results. Making a program scale over multiple processors isn't easy.
Sean O'Rourke entry in the wide finder benchmark (http://www.cs.ucsd.edu/
~sorourke/wf.pl) offers an interesting approach to this, though it isn't
exactly optimized for readability.

Regards,

Leon Timmermans
.

Relevant Pages

  • Re: For the same cost as the cheapest Macbook Pro I can get
    ... Memory 3GB System Memory ... Hard Drive Type SATA (7200 rpm) 320GB ... Graphics Intel® Graphics Media Accelerator ... Grand Central was not the first way that multiple processors were ...
    (comp.sys.mac.advocacy)
  • Re: Memory could not be read errors
    ... exact error message has been reported on the web for a completely unrelated ... The address is closest to two open DLLs in memory, ... on a laptop computer but not a the dual processor. ... threading errors. ...
    (microsoft.public.dotnet.general)
  • Re: perl multithreading performance
    ... and about as much times more CPU. ... The speed of perl's threading is dependent on how much you share between ... to divide the work first, and only used shared resources to report back ... Making a program scale over multiple processors isn't easy. ...
    (comp.lang.perl.misc)
  • Re: Reasons to choose CLISP over other free implementations
    ... Better memory management. ... of static content to Apache and dynamic content is relatively quick to ... But today when implementations exist who support hundreds of thousands ... I want to see actor based threading like before ...
    (comp.lang.lisp)
  • Re: About volatile qualifier
    ... memory as well, so it should be valid for purposes of sharing data ... threading systems" and their issues such as the need to use memory ... WRT adding memory barrier semantics to volatile access and ordering ...
    (microsoft.public.vc.language)