Re: optimize log parsing

"it_says_BALLS_on_your forehead" <simon.chao@xxxxxxx> wrote:
> xhos...@xxxxxxxxx wrote:
> >
> > BTW, I'm curious about the bottleneck in your code. If your code is
> > CPU-bound, then parallelization to 20 processes won't help much unless
> > you have 20 CPUs. If it is disk-drive bound, then parallelization won't
> > help unless your files are on different disks (and probably on
> > different controllers.)
> >
> > Xho
> >
>
> ahh, that makes sense, thanks!
>
> to answer your question, i'm working on a box with 16 CPUs. the number
> 20 is from code that i inherited from a predecessor. there used to be
> 10 processes, and he changed it to 20, and it went faster, so 20 it
> stayed. should i change it to 16?

There is no real reason to use a max of 20 (or anything else greater than
16) if you only have 16 CPUs, but using 20 shouldn't hurt, either. There
may be slightly more kernel overhead with 20 than 16, but on modern OSes
the difference will be negligible.

One way it could matter is if other process are also using the machine.
For example, if there are 8 other CPU-bound processes (all of same nice
level as yours), then if you use max of 16 you will get ~16/24 of the CPU
time and the other programs will get ~8/24, while if you use a max of 20,
then you will get ~20/28 of the time and they will get ~8/28.


> also, what's the difference between using Parallel::ForkManager to do
> 20 tasks, and looping through system('script.pl &') 20 times?

In that case (when the number of tasks to do is less than or equal to the
maximum number of processes you want running at one time) the biggest
difference is that ForkManager forks a copy of the currently running
perl interpreter, which is very fast, while system("foo.pl&") will cause a
brand new interpreter to start up, which is a lot slower than forking a
copy.[1] (Although both of these are fast enough it doesn't really matter
if you are only doing it 20 times. (and if you are doing it far more than
20 times, than the initial assumption of this paragraph is not met))

Of course, this also means that the new interpreter started with "system"
is independent of the old one. This could be good (i.e. its name space
is not contaminated, it doesn't corrupt the parent's database handles, its
file locks are independent from the parent's etc.) but is at least as
likely to be bad (it has to redo the "use"s and reparse any input that
might already have been "use"d or parsed in the original interpreter, it
doesn't readily inherit its parent's file locks, etc.)

> i mean, i
> see an advantage in that with ForkManager, when one processes dies,
> another takes its place so you don't need to pre-ordain which process
> does which work.

But of course that only matters if there are more tasks to run than there
are simultaneous processes you wish to allow. ForkManager is great if you
want to run 400 tasks, but want to have only 20 going at any one time. If
you only want to run 20 in the first place and will do them all at the same
time, that removes a lot (but not all) of the benefit of using ForkManager.

> but let's assume that each process has exactly the
> same amount of work and processes that work with the same speed. would
> ForkManager be faster?

Probably, but only because simple forking is faster than "system".

> Is there ever a case where multiple system()
> calls is the answer?

If the child processes spawned by ForkManager have to call "system" or the
moral equivalent anyway (for example, because the task you need to run is
an external executable, not Perl code. Or it is Perl code but you need to
run it in a standalone interpreter for some reason) then there is no reason
to combine both ForkManager and system, just go with system. But again,
this only applies if you don't need to throttle the number of simultaneous
processes.

If the tasks you need to do are very short and you need to do a very large
number of them (400 is *not* a very large number), you should try to batch
them up so that each process/thread does bunch of them. This is a basic
matter of overhead, and is largely true regardless of whether you use
ForkManager, system, threads or something else.


Xho


[1] Actually, "system" will internally first fork a copy of the current
interpreter, and then do an exec, which discards this forked copy and
causes a new intepreter to be started. At least on Linux/Unix, which I
suspect you are using as I've never seen a 16 CPU Windows machine.

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
.

Relevant Pages

  • Re: optimize log parsing
    ... then using ForkManager is probably not the best choice. ... The equivalent thread implementation to the above fork ... Parallelization is inherently an optimization step. ... That was on a single CPU machine. ...
    (comp.lang.perl.misc)
  • Re: optimize log parsing
    ... > then using ForkManager is probably not the best choice. ... >> and compare it to an equivalent threads-implementation where the items ... The equivalent thread implementation to the above fork ... > Parallelization is inherently an optimization step. ...
    (comp.lang.perl.misc)