Re: Wait for background processes to complete

On Jan 14, 2:48 pm, xhos...@xxxxxxxxx wrote:
pgodfrin <pgodf...@xxxxxxxxx> wrote:
On Jan 14, 11:11 am, xhos...@xxxxxxxxx wrote:

The fork returns (to the parent) the pid of the process forked off.
(but you don't actually need to know the pid if you merely want to
wait, rather than waitpid.) If that forked-off process then itself
starts the cp in the background, of course you are no better off. But
if the forked-off process either becomes cp (using exec) or it starts
up cp in the foreground (using system without a "&"), then you now have
something to wait for. In the first case, you wait for cp itself. In
the second case, you wait for the forked-off perl process which is
itself waiting for the cp.

$ perl -wle 'use strict; fork or exec "sleep " . $_*3 foreach 1..3 ; \
my $x; do {$x=wait; print $x} until $x==-1'
438
439
440
-1

Hi Xho,
Well - your code and concepts work fine when you want to wait
sequentially.

Is there an alternative to waiting sequentially? Waiting sequentially
is what the shell does, too, behind the scenes.

My goal here is to fire off x number of process and then
wait for ALL of them to complete

That is what my code does.

(this is basically rudimentary job
control, trying to use the shell concepts and maximize parallelism).
So, that requires the use of the & to send a process to the
background.

No, that is not required. Since Perl is not a shell, there really
isn't such a thing as the "background" in a Perl context. However,
fork will launch another process which runs without blocking the original
process (well, at least not until the original process asks to be blocked)
or blocking sibling processes. That is what you want, no?

It looks like exec doesn't recognize the &, which leaves
the system() command - which leaves us back at square one. The shell
completes and the program has nothing to wait for.

Yes, so don't do that. "&" basically means "Fork and then don't wait".
Since that isn't what you want to do, then don't do that. Do your own
fork, and then do your own wait.

Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Hi Xho,
It seems to me that firing off say 5 processes with the '&' character
to send them to the background is 5 parallel processes, while
executing them off one at a time is sequential. Your code (fork or
exec "sleep" ... ) waits for each sleep process to complete - so that
is what I meant by "waiting sequentially". Technically speaking you're
right - but the idea is to have tasks run in parallel versus
sequentially, which is ostensibly faster.

Insofar as using fork() - your original observation still stands -
system() runs a shell command and then terminates after sending the
command within to run (I watched two different PIDs - the long running
command within the system() statement continued while the shell became
<defunct>)

To make a long story short, this is how I solved the problem. It seems
that the PIDs of the commands run via system() and the '&' background
thing end up belonging to the same Process Group - seen in the ps
command, plus a little extra:

ps -C cp -o pid,pgid,command
PID PGID COMMAND
29068 29063 cp example01.txt example01.txt.old
29070 29063 cp example02.txt example02.txt.old
29072 29063 cp example03.txt example03.txt.old
29074 29063 cp example04.txt example04.txt.old
29076 29063 cp example05.txt example05.txt.old
29078 29063 cp example06.txt example06.txt.old

So I wrap the ps command and do some looping:

for (;;)
{
open PGRP, "ps -C cp h |" ;
@pidlist=<PGRP> ;
if ($#pidlist<0) {die "\nNo more processes\n" ;}
}

It's not pretty but it works...

But, I believe this is an architectural FLAW with Perl.

regards,
pg






.

Relevant Pages

  • Re: Fork, exec - setsid?
    ... So I came up with the need to fork some processes away ... > of those or the spawned shell. ... The shell will have the pid that forkreturned to the parent. ... > How do I kill the forked away programms without knowing their PID? ...
    (comp.lang.perl.misc)
  • Re: Complete process information
    ... ps eauwww {pid} While an exact Solaris equivalent ... > command line a process was started with. ... First is to check your shell. ...
    (comp.unix.solaris)
  • Re: pipin problems - stdin & stdout messed up after executing piped command
    ... > function to implement piping. ... > the problem is that everytime i call a piped command, ... Note that you fork first, then pipe, then fork, rather than pipe, fork, fork. ... Of course on of the things that made ksh a more efficient shell than the ...
    (comp.unix.shell)
  • Re: running a db2 script using execl
    ... >> I have a C program in which I use the fork a sub shell. ... > command to see if you get different output. ... > need to use execl, since it requires you to call fork(), etc. ... and "LOAD" command for db2. ...
    (comp.unix.aix)
  • Re: recursive grep problem [zip logs]
    ... fork per file (assuming your shell is recent enough that echo is an ... (weird find command, btw: is your directory really called 't'?) ... Avoiding forking can easily lead to thousandfold speedups in shell ... scripts, even on platforms, like Linux, where fork() is fast. ...
    (uk.comp.os.linux)