Re: perl multithreading performance

On Aug 27, 4:39 pm, Leon Timmermans <faw...@xxxxxxxxx> wrote:
On Wed, 27 Aug 2008 12:59:36 -0700, dniq00 wrote:

Everything works fine, HOWEVER! It's all so damn slow! It's only 10%
faster than single-process script, consumes about 2-3 times more memory
and about as much times more CPU.

I've also tried abandoning the Thread:Queue and just use threads::shared
with lock/cond_wait/cond_signal combination, without much success.

I've tried to play with $cpu_count and $buf_size, and found that after
$buf_size > 1000 doesn't make much difference, and $cpu_count > 2
actually makes things a lot worse.

Any ideas why in the world it's so slow? I did some research and
couldn't find a lot of info, other than the way I do it pretty much the
way it should be done, unless I'm missing something...

Hope anybody can enlighten me...

THANKS!

The speed of perl's threading is dependent on how much you share between
threads. Sharing the lines before processing them can become a
bottleneck, I suspect that's the problem in your case. You probably want
to divide the work first, and only used shared resources to report back
the results. Making a program scale over multiple processors isn't easy.
Sean O'Rourke entry in the wide finder benchmark (http://www.cs.ucsd.edu/
~sorourke/wf.pl) offers an interesting approach to this, though it isn't
exactly optimized for readability.

Regards,

Leon Timmermans

Thanks for the link - trying to figure out whattahellisgoingon
there :) Looks like he's basically mmaps the input and begins reading
it starting at different points. Thing is, I'm using <> as input,
which can contain hundreds of gigabytes of data, so I'm not sure how's
that going to work out...

.

Relevant Pages

  • Re: perl multithreading performance
    ... consumes about 2-3 times more memory ... The speed of perl's threading is dependent on how much you share between ... Making a program scale over multiple processors isn't easy. ...
    (comp.lang.perl.misc)
  • Re: Teaching old threaded programmers new parallel programming tricks
    ... And it is hard to retrofit threading into existing complex code. ... less deterministic makes them significantly harder to debug in many ... One always needs to get code working reliably on a single thread first, ... multiple processors. ...
    (comp.programming.threads)
  • Re: PC building tips?
    ... and threading, although you DO need multitasking or threading to take advantage ... of multiple processors. ... applications that can work well on a single CPU. ...
    (rec.audio.pro)
  • Re: PC building tips?
    ... and threading, although you DO need multitasking or threading to take advantage ... of multiple processors. ... applications that can work well on a single CPU. ...
    (rec.audio.pro)