trying to understand fork and wait

From: John (jguad98_at_hotmail.com)
Date: 11/20/03 

Date: 19 Nov 2003 16:15:21 -0800

I attempted to post yesterday, browser crapped out without returning,
don't know if message got out alive ... therefore, my apologies if I
have double posted.

I would really appreciate some feedback on a Perl program I'm trying
to write ... I know what I want it to do, but I'm not certain I'm
going about it the right way.

[requirement]
monitor application logfiles (multiple) on unix server (singular) for
error messages, re-write the error messages to syslog >>as they
occur<<.

[background / challenges]
the application is stupid & closed, cannot be modified, so I am trying
to handle this with a Perl script. The target application has upwards
of 100 users that can be online concurrently. Each user session
writes a user log to an individual directory & logfile. Meaning,
upwards of 100 logfiles all being written to at the same time. There
is no defined start or stop time for any given user, so far no known
signal to indicate that user session has started. Only way (so far)
to tell that a user session is active is to check to see if the log
file has been created (they overwrite the same file every time a
session starts) and is being written to.

My plan (so far) is to have a single Perl script running while the
application is up, periodically scanning (via a simple loop & sleep)
to detect logfiles. Upon detection of active logfile, spawn a child
to do the reading & error message trapping. This so that I have one
program keeping track of what logs are a) present, b) active, and c)
being monitored; then for each log, a child program would sit on an
open filehandle reading it and comparing messages found to a known
list of errors, and when an error is detected send it to syslog via
the logger command.

I know that my design could potentially put a load on the box, but it
is a big & strong fellow ... I think he can take it. But I have never
used fork before, have seen no understandable sample scripts to draw
inspiration from, and even after spending several days reading
articles from clpmisc and clpmod, as well as the Perl Cookbook and
Programming Perl, I still do not understand how to use fork and wait
and related functions properly to accomplish my task.

Please note I would be accurately described as a novice programmer so
there may be some poorly written code. Also also note that some
desired functionality has not been written yet (main loop, sleeps & a
way to track the children) because I'm still trying to get the core
functionality designed properly.

So without further ado, here is my code presented for attack ... er, I
mean analysis and discussion:

#!/usr/local/bin/perl
#=================================================================#
#-----------------------------------------------------------------
# some global vars
#-----------------------------------------------------------------
  $mypid=$$;
  $basedir="/paf/rpas";
  @domainlist=`ls $basedir | egrep '^(top|key)..$'`;
  $patternfile="/path/to/patterns.list";
  open(PATTERNS,$patternfile);
  @patternlist=<PATTERNS>; # this is my store of error messages to
watch for
  close(PATTERNS);
#-----------------------------------------------------------------
# scan the directories
#-----------------------------------------------------------------
  foreach $domain (@domainlist) {
    chomp($domain);
    $userdirbase="$basedir/$domain/users";
    opendir(USERS, "$userdirbase");
    #-------------------------------------------------------------
    # read the dir
    #-------------------------------------------------------------
    while ($file = readdir(USERS)) {
        #---------------------------------------------------------
        # only select files matching the username pattern
        #---------------------------------------------------------
        next unless $file =~ /^[a-z][a-z]\d\d\d\d\d$/;
        #---------------------------------------------------------
        # check if we have a user dir
        #---------------------------------------------------------
        $currentuserdir="$userdirbase/$file";
        if (-d "$currentuserdir") {
            #-----------------------------------------------------
            # check if there is an applog in the user dir
            #-----------------------------------------------------
            $logfile="$currentuserdir/applog";
            if ( -e "$logfile") {
                #-------------------------------------------------
                # is the log open or closed?
                #-------------------------------------------------
                $return=`grep 'known end of session message'
$logfile`;
                #-------------------------------------------------
                # if open, fire the monitor
                #-------------------------------------------------
                if (! $return) { # the grep failed, ergo the file is
active
                  #-------------------------------------------------
                  # line below is because everything I've read about
fork
                  # says I gotta wait on the child pid (?) or it will
                  # become a zombie which is bad. I don't understand
                  # what this does or how it works.
                  #-------------------------------------------------
                  $SIG{'CHLD'} = sub { wait(); };
                  #-------------------------------------------------
                  # now we fork if and only if we are the original
program
                  # we consider ourselves too young to have
grandchildren
                  #-------------------------------------------------
                  if ($$ == $mypid) {
                    $kidpid = fork() or die "cannot fork: $!";
                  } else {
                    #-------------------------------------------------
                    # are we in the child process?
                    #-------------------------------------------------
                    if ($kidpid == 0) {
                      #---------------------------------------------
                      # read the user log
                      #---------------------------------------------
                      open(LOG,"$file");
                      while ($line = <LOG>) {
                        close(LOG), exit if ($line =~ /known EOS
message/);
                        foreach $patt (@patternlist) {
                          if ($line =~ /$patt/) {
                            $now=localtime(time);
                            system("logger $now $domain $user $line");
                          }#end child's if pattern match
                        }#end child's foreach loop
                      }#end child's while loop
                    }# this is the end of the child block
                  }#end if mypid
                }#end if $return
            }#end if logfile exists
        }#end if directory (go to top of parent while loop)
    } #end parent while loop (go to top of parent foreach loop)
    closedir(USERS);
  }#end parent foreach loop
#-----------------------------------------------------------------
  exit 0;
#== EOF ==========================================================#

The parts that I am worried about are the "wait" (the line was cribbed
in it's entirety from a clpmisc article) and the "fork" (based on
another clpmisc article, but modified to do what I think I want it to
do).

What is it about the wait that I don't get? Well ... I've seen
articles discussing using system() versus fork() and they tell me that
system() has a built-in wait ... and I know that I cannot use system()
because the calling script (parent?) will not continue
processing/looping until the system() function returns ... I do not
want to wait for a return ... I want to keep going in my loop.

Will "wait" make me wait for a return or not? I have read the
perldoc, manpage, Programming Perl, the Perl Cookbook and multiple
clpm articles, and it still is not clear to me what happens when I
invoke / call / issue the "wait" function. My whole script is
predicated upon the notion that I can keep looping and forking
children willy-nilly ... that is to say, "fire and forget" but also
keep an eye on the children insofar as I'd like to track how many I
have and what logs are being monitored.

Then with regard to the fork ... every example I've seen shows a block
for the child along with a block for the parent all under the fork
function, which was very confusing because I always thought that the
parent was the script which generated the child (i.e., OUTside the
fork, not INside), until I read a description saying that the child is
actually a copy of the entire script and apparently can be limited to
execute only certain lines by testing the pid in an 'if' block.

So, in my child image I know that I do not want it to ever execute a
'parental' instruction, so you can see above that I did not actually
code a 'parent' block under the fork function unlike every single
example I've seen so far. Why am I the only person to not have a
'parent' block? There must be a reason for that parent block, but I
cannot figure out what it is ... again, I have read the relative
perldocs, manpages, and Perl book entries. I just don't get it.

I have not actually run the above program mostly because I don't
understand how the wait & fork will work and I fear unintended
consequences. I'm planning on building a test case, but in the
meanwhile I would be quite grateful for any explanations as to things
I am doing right or wrong in my program, as well as any suggestions on
a better way to solve my problem.

regards,

John G.

Relevant Pages

  • Re: understanding shared memory
    ... fork() doesn't come into the picture for subroutines. ... > a separate SpamAssassin server and feeding the email to it from the ... with multiple threads in Perl, ... script didn't die on a slower single-CPU machine. ...
    (comp.os.linux.misc)
  • Problem with Parallel::ForkManager on Windows Vista
    ... to register as completed with the waitpid within Perl nor does it ... decrement the parent process's thread count in the process list. ... I'm aware that fork isn't fully supported really on Windows but maybe ...
    (comp.lang.perl.misc)
  • Re: Sending email using perl
    ... > I use the folowing script to send email directly to a server (using its ip ... the parent or child process. ... both the parent and child will ... fork other children, which will also attempt to send an e-mail. ...
    (comp.lang.perl.misc)
  • Re: Forking and subshells - Keeping the parent active
    ... Running a bash script, that is setup to fork a subshell ... Want the parent to keep running while the child is doing its processing. ...
    (alt.os.linux.suse)
  • Forking and subshells - Keeping the parent active
    ... Running a bash script, that is setup to fork a subshell ... Want the parent to keep running while the child is doing its processing. ...
    (alt.os.linux.suse)