Parallel::ForkManager and Net::FTP problem?



Hello, I am running the following code:
#------
#!/apps/webstats/bin/perl

use strict; use warnings;
use Net::FTP;
use File::Copy;
use Parallel::ForkManager;

require "tools.pl";

my $area = shift;

my $processDate;
$processDate = '051006';

my $command = "ssh -l freclogs eclog31
/export/home/freclogs/simon/1-perl/getFileSize.pl $area";

my @result = `$command`;
my %fetch;

for (@result) {
chomp;
my ($key, $value) = split('=>');
$fetch{$key} = $value;
}

my $toList = "simon.chao\@fmr.com";

&startCycle( $area, 16, \%fetch );

#--- subs ---
sub startCycle {
my ($area, $num_groups, $data_ref) = @_;
my %data = %{$data_ref};

my $ftp;
my $server = 'eclog31';
unless ($ftp = Net::FTP->new($server)) {
doMail("Problem with $0", "Can't connect to $server with ftp,
$@\n", $toList);
die;
}

my ($userName, $password) = ('user','password');

unless ($ftp->login($userName, $password)) {
doMail("Problem with $0", "Can't login to $server with ftp using
-$userName- and -$password- $@\n", $toList);
die;
}
$ftp->binary();

my %missing;

my $pm = Parallel::ForkManager->new($num_groups);

while( my ($file, $size) = each %fetch ) {
$pm->start and next;

if ($size) {
# fetch and parse
my ($server, $dir, $file_type, $slice, $controlNo, $isCritical) =
split(/\|/, $file);

my $remoteFile = "$dir/$processDate.gz";
my $localFile =
"$cycleType{$area}->{rawLogs}/$file_type.$processDate$slice";

next if (-e $localFile);
if ($ftp->get($remoteFile, $localFile)) {
print "got $remoteFile to $localFile\n";
my $doneFile = $localFile;
$doneFile =~ s/^.*\///g;
$doneFile =~ s/\.gz$//;
$doneFile = "$cycleType{$area}->{work}/$doneFile";
# Kick Off Parsing for this file:
my $command = "preParse.pl $processDate $localFile $doneFile
$hash{$area}->{parseMethod}";
system($command);
}
else {
print "FTP MESSAGE: $ftp->message\n";
}
}
else {
# Capture missing logs to deal with later
# can add to a hash, or write to a file, or maybe even use
Storable.
# maybe sleep 300, then kick off another startCycle?

print "failed to get $file\n";
$missing{$file} = localtime();
}

$pm->finish;
}
$ftp->quit();

my $mailBody = "";
while( my ($missingFile, $time) = each %missing ) {
$mailBody .= "File $missingFile missing at $time\n";
}

print "$mailBody\n";
doMail("Subject", "$mailBody", $toList);
$pm->wait_all_children;
}

#-----

and the output is:

[smro180 191] /webstatmmk1/pre/authlogs/bin > betaProcess.pl authlogs &
[1] 20047
[smro180 192] /webstatmmk1/pre/authlogs/bin >

*************************************************************************
* WARNING: This system is restricted to authorized users for
legitimate *
* business purposes and is subject to audit. The unauthorized access,
*
* use, or modification of this system or of the data contained
therein *
* or in transit to/from it is a criminal violation of Federal and
*
* state laws
*

*************************************************************************
Argument "" isn't numeric in numeric comparison (<=>) at
../betaProcess.pl line 101.
eclog31|/logs/raw/fswas103/FebSec.febsec.audit|fsaudit.fidelity|.fswas103.gz

39214498
eclog31|/logs/raw/fswas101/FebSec.febsec.audit|fsaudit.fidelity|.fswas101.gz

38977495
eclog31|/logs/raw/fswas102/FebSec.febsec.audit|fsaudit.fidelity|.fswas102.gz

38807337
eclog31|/logs/raw/fswas401/FebSec.febsec.audit|fsaudit.fidelity|.fswas401.gz

37928934
eclog31|/logs/raw/fswas402/FebSec.febsec.audit|fsaudit.fidelity|.fswas402.gz

37771574
eclog31|/logs/raw/fswas403/FebSec.febsec.audit|fsaudit.fidelity|.fswas403.gz

37770895
eclog31|/logs/raw/fswas301/FebSec.febsec.audit|fsaudit.fidelity|.fswas301.gz

37762543
eclog31|/logs/raw/fswas303/FebSec.febsec.audit|fsaudit.fidelity|.fswas303.gz

37707907
eclog31|/logs/raw/fswas302/FebSec.febsec.audit|fsaudit.fidelity|.fswas302.gz

37608154
eclog31|/logs/raw/fswas102/FebSec.ibg.audit|fsaudit.ibg|.fswas102.gz

4062110
eclog31|/logs/raw/fswas103/FebSec.ibg.audit|fsaudit.ibg|.fswas103.gz

4056833
eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz

# the above ^^^ is fake, to test

FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
failed to get
eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
Unable to close datastream at ./betaProcess.pl line 174



....i apologize for the copious amount of text, but i wanted to present
the problem as fully as possible.

there were even times when i got this message:
Cannot start another process while you are in the child process at
/apps/webstats/lib/perl5/site_perl/5.6.1/Parallel/ForkManager.pm line
281.


....that may have been because i did not kill all of the processes?
i tried ps -ef | grep beta
and did a kill 9 <pid> on all of them, but i'm not sure if i have to
wait for them all to die. even after i ps -ef | grep beta again, and
see that there are no pids associated with betaProcess.pl, i have a
suspicion (perhaps unwarranted) that the processes are still stubbornly
clutching on to life.

i thought this was going to be relatively simple to implement--but
after doing some research online i could not discover how to resolve
these issues.

basically i'm trying to fork some processes and FTP get some files,
after i get them, i call a script preParse.pl via a system() call. this
may be causing some difficulty, but i'm too new with
Parallel::ForkManager to have experience with this. if you've even read
to this point, you've impressed me :-). i hope someone can give me some
hints.

.