Re: Expect - memory leak issue
- From: JD <ChengTeh.chen@xxxxxxxxx>
- Date: Wed, 10 Sep 2008 08:16:32 -0700 (PDT)
On Sep 10, 3:09 am, cla...@xxxxxxxxx (Cameron Laird) wrote:
In article <t9xxk.5$Fm...@xxxxxxxxxxxx>,
Gerald W. Lester <Gerald.Les...@xxxxxxx> wrote:
JD wrote:
Hi,
We are using expect 5.43.0. And we found there might be potential
memory leak issue for this version.
In our application, we try to use expect to automatic ftp pull files
from other server every 5 minitues as a daemon process.
Below is one of the monitoring result every 20 minutes via ps command
in our server. In the column "Share(KB)", the memory consumption is
increasing.
Could anyone help me to confirm whether this is memory leak issue? How
to resolve it except restart the process again and again?
Well, you do not you do not need Expect for this. You could use plain Tcl
and the ftp module in TclLib.
Assuming the memory leak is really in Expect and not your code, then the
above approach would solve your memory leak problem.
.
.
.
It's almost certain that your best path forward is to
abandon Expect and pull the FTP items some other way.
I like Expect, and use it daily; it's probably not a
good fit for your situation, though.
What you describe should be only a few minutes of coding.
I'd happily do it for a peck of avocados (or shoes for
the children--I'm trying to think of some adequately
vivid way to express the magnitude of the effort).- Hide quoted text -
- Show quoted text -
Hi all,
Below is the script we had for ftp_pull and I suspect this script or
expect has memory leak issue.
#!/opt/CVGSxpect/bin/expect --
#!/opt/CVGSxpect/bin/expect --
#!/usr/local/bin/expect --
#!/tools/sun5/cygnus/H-sparc-sun-solaris2/bin/expect --
#!/apps/tcl-8.0.3/bin/expect --
set PROGNAME ftp_pull
set USAGE "usage - $PROGNAME \[OPTIONS] ne_name ip_addr login passwd
remote_dir file_spec poll_int
OPTIONS:
-M n set maximum number of files pulled per ftp connection to n
-A pull remote files in alphanumeric order (default) (exclusive
to -T -s)
-T pull remote files in chronological order (exclusive to -A -s)
-s pull remote files in random order (exclusive to -A -T)
-a force FTP transfers to ASCII mode (exclusive to -b)
-b force FTP transfers to binary mode (exclusive to -a)
-d delete remote files after ftp retrieval (default) (exclusive
to -m -r)
-m dir move remote file to remote directory dir (exclusive to -d -r)
-r spec rename remote file according to spec . (exclusive to -d -m)
spec has format /pat1/pat2/ where pat1 describes the
filename
pattern to change and pat2 describes the filename pattern
to replace
-N prepend the local filename with the NE name on the command
line
-t sec the maximum timeout seconds the program will wait for an ftp
command
response. (default 300 secs)
-vms perform special processing for pulling files from VAX/VMS
systems. No
remote and local file length checking performed.
-D n set debug trace level to n
-W to specify that the remote machine is running in windows OS
ne_name the NE name collecting the retrieved files
ip_addr the IP address of the remote network element
login the login name for the remote network element
passwd the password for the remote network element
remote_dir the remote directory to check for files
file_spec the pattern describing the remote files to retrieve
poll_int the duration in minutes to wait between ftp sessions
"
#-------------------------------------------------------------------------------
# This is a debugging function utilized throughout the script
#-------------------------------------------------------------------------------
proc debug {level msg} {
global DEBUG
if { $level<=$DEBUG } { send_user "DEBUG($level) - $msg\n" }
}
#-------------------------------------------------------------------------------
# This function provides a uniform method fspecor writing time stamped
messages to
# the daily log files. A new log file is created each day.
#-------------------------------------------------------------------------------
proc log_msg {MSG} {
global LOGDIR LOGFILE THEN PROGNAME NE_NAME
# Create a current datestamp string
set NOW [exec date +%m%d%Y]
if { ![string match $NOW $THEN] } {
# Create the name of the current log file
set FILE "$PROGNAME\_$NE_NAME\_$NOW.log"
debug 2 "opening log file $FILE"
set LOGFILE "$LOGDIR/$FILE"
# disable any previous open log file first
log_file
# open the new log file
log_file $LOGFILE
# Copy the NOW value to THEN
set THEN $NOW
}
if { [string length $MSG]>0 } {
#
# if a msg was supplied then log the message to the file
#
send_log "[exec date +%T] $MSG\n"
} else {
#
# Put a start up banner in the log file to show that the script is
running
#
set STARS "***************************************"
send_log "\n"
send_log "#$STARS$STARS\n"
send_log "# $PROGNAME started at [exec date]\n"
send_log "#$STARS$STARS\n"
}
return 0
}
#-------------------------------------------------------------------------------
# This is global debugging procedure that is called by the setflags
procedure.
# It sets tracing and logging features in the Tcl processor.
#-------------------------------------------------------------------------------
proc tcl_debug {} {
global DEBUG
if { $DEBUG>2 } { log_user 1 } else { log_user 0 }
if { $DEBUG>3 } { exp_internal 1 } else { exp_internal 0 }
}
#-------------------------------------------------------------------------------
# This function parses the option flags encoded either on the ftp_pull
command
# line or in the NE_OPTIONS column of the NE_FTP_CONFIG database
table. The parsed
# flags are stored in the global array FLAGS indexed by NE name. The
global
# command line flags are stored in the FLAGS array indexed under
"_command_line_".
#-------------------------------------------------------------------------------
proc parseflags {ne_name flags} {
global FLAGS USAGE
global GROUP_MODE
global DEBUG
# initialize the number of arguments on the flags string
set flgc [llength $flags]
for {set flgi 0} {$flgi<$flgc} {incr flgi} {
set flg "[lindex $flags $flgi]"
if { [string match $flg "--"] } {
break
} elseif { [string index $flg 0] != "-" } {
break
} elseif { [string match $flg "-D"] } {
# the -D option my only be used on the ftp_pull command line
if { ![string match "_command_line_" $ne_name] } {
send_user "the -D option may only be used on the ftp_pull command
line\n"
return -1
}
#
# DEBUG 0 - for production, turns off all debugging output
# DEBUG 1 - turns on the tracing of procedure calls
# DEBUG 2 - turns on the tracing of key variables
# DEBUG 3 - turns on expect script dialogs
# DEBUG 4 - turns on expect pattern tracing feature
#
incr flgi
set FLAGS($ne_name,DEBUG) "[lindex $flags $flgi]"
set DEBUG "[lindex $flags $flgi]"
} elseif { [string match $flg "-M"] } {
incr flgi
set FLAGS($ne_name,MAXPULLS) "[lindex $flags $flgi]"
} elseif { [string match $flg "-A"] } {
set FLAGS($ne_name,PULL_ORDER) "A"
debug 2 "PULL_ORDER option set to A"
} elseif { [string match $flg "-T"] } {
set FLAGS($ne_name,PULL_ORDER) "T"
debug 2 "PULL_ORDER option set to T"
} elseif { [string match $flg "-s"] } {
set FLAGS($ne_name,PULL_ORDER) "R"
debug 2 "PULL_ORDER option set to R"
} elseif { [string match $flg "-d"] } {
set FLAGS($ne_name,FILE_DISPO) "d"
} elseif { [string match $flg "-m"] } {
set FLAGS($ne_name,FILE_DISPO) "m"
incr flgi
set FLAGS($ne_name,MOVEDIR) "[lindex $flags $flgi]"
} elseif { [string match $flg "-r"] } {
set FLAGS($ne_name,FILE_DISPO) "r"
incr flgi
set FLAGS($ne_name,RENAMESPEC) "s[lindex $flags $flgi]"
} elseif { [string match $flg "-N"] } {
set FLAGS($ne_name,PREPEND_NENAME) 1
} elseif { [string match $flg "-a"] } {
set FLAGS($ne_name,FTP_MODE) "ascii"
} elseif { [string match $flg "-b"] } {
set FLAGS($ne_name,FTP_MODE) "binary"
} elseif { [string match $flg "-t"] } {
incr flgi
set FLAGS($ne_name,TIMEOUT) "[lindex $flags $flgi]"
} elseif { [string match $flg "-vms"] } {
set FLAGS($ne_name,REMOTE_OS) "vms"
} elseif { [string match $flg "-W"] } {
set FLAGS($ne_name,REMOTE_OS) "windows"
} elseif { [string match $flg "-G"] } {
# the -G option my only be used on the ftp_pull command line
if { ![string match "_command_line_" $ne_name] } {
send_user "the -G option may only be used on the ftp_pull command
line\n"
return -1
}
set GROUP_MODE 1
} else {
send_user "invalid option detected, \"$flg\"\n"
return -1
}
}
return $flgi
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc initflags {} {
global FLAGS
# reset the whole FLAGS array
unset FLAGS
# initialize the default values for each flag
set FLAGS(_flag_default_,IP_ADDR) ""
set FLAGS(_flag_default_,LOGIN) ""
set FLAGS(_flag_default_,PASSWD) ""
set FLAGS(_flag_default_,RMT_DIR) ""
set FLAGS(_flag_default_,FILE_SPEC) ""
set FLAGS(_flag_default_,POLL_INT) ""
set FLAGS(_flag_default_,DEBUG) 0
set FLAGS(_flag_default_,MAXPULLS) 1000000
set FLAGS(_flag_default_,PULL_ORDER) "A"
set FLAGS(_flag_default_,FILE_DISPO) "d"
set FLAGS(_flag_default_,MOVEDIR) ""
set FLAGS(_flag_default_,RENAMESPEC) "s///"
set FLAGS(_flag_default_,PREPEND_NENAME) 0
set FLAGS(_flag_default_,FTP_MODE) ""
set FLAGS(_flag_default_,REMOTE_OS) "unix"
set FLAGS(_flag_default_,TIMEOUT) 300
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc setflags {ne_name} {
global FLAGS
global NE_NAME
global IP_ADDR LOGIN PASSWD RMT_DIR FILE_SPEC POLL_INT
global DEBUG MAXPULLS PULL_ORDER FILE_DISPO MOVEDIR RENAMESPEC
global PREPEND_NENAME FTP_MODE REMOTE_OS TIMEOUT
set flist {
IP_ADDR
LOGIN
PASSWD
RMT_DIR
FILE_SPEC
POLL_INT
MAXPULLS
PULL_ORDER
FILE_DISPO
MOVEDIR
RENAMESPEC
PREPEND_NENAME
FTP_MODE
REMOTE_OS
TIMEOUT
DEBUG
}
set cnt [llength $flist]
debug 2 "<<<<<<<<<<<<<<< NE_NAME=$ne_name >>>>>>>>>>>>>>>"
for {set f 0} {$f<$cnt} {incr f} {
set flag "[lindex $flist $f]"
if { [info exists fval] } { unset fval }
if { [info exists FLAGS($ne_name,$flag)] } {
set fval $FLAGS($ne_name,$flag)
set $flag $fval
debug 2 "$flag=$fval - from $ne_name"
} elseif { [info exists FLAGS(_command_line_,$flag)] } {
set fval $FLAGS(_command_line_,$flag)
set $flag $fval
debug 2 "$flag=$fval - from command line"
} elseif { [info exists FLAGS(_flag_default_,$flag)] } {
set fval $FLAGS(_flag_default_,$flag)
set $flag $fval
debug 2 "$flag=$fval - from default"
} else {
set $flag ""
debug 2 "$flag=NULL - NULL VALUE SET"
}
}
# Split the RMT_DIR string into a list
set rdirs "[split $RMT_DIR ";"]"
set RMT_DIR ""
foreach rdir $rdirs {
if { ![string match "{}" $rdir] } { lappend RMT_DIR $rdir }
}
debug 2 "RMT_DIR=[list $RMT_DIR]"
# Split the FILE_SPEC string into a list
set specs "[split $FILE_SPEC ";"]"
set FILE_SPEC ""
foreach spec $specs {
if { ![string match "{}" $spec] } { lappend FILE_SPEC $spec }
}
debug 2 "FILE_SPEC=[list $FILE_SPEC]"
tcl_debug
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc parsecommandline {} {
global FLAGS
global NE_NAME GROUP_MODE GROUP_MEMBERS GROUP_COUNT
global argv
# initialize the number of arguments on the command line
set argc [llength $argv]
# parse off any command line options and store them in the FLAGS
array
set argi [parseflags _command_line_ $argv]
if { $argi+1 > $argc } { return 1 }
# Grab the NE_NAME off of the command line
# NOTE - the NE_NAME may contain an NE name or a GROUP name
set NE_NAME [lindex $argv $argi]; incr argi
# set up the output log file and print a startup banner in it
log_msg ""
# If this a normal single NE and non database type ftp_pull command
then
# parse all arguments off of the comamnd line.
if { $argi+5 < $argc } {
set FLAGS($NE_NAME,IP_ADDR) "[lindex $argv $argi]"; incr argi
set FLAGS($NE_NAME,LOGIN) "[lindex $argv $argi]"; incr argi
set FLAGS($NE_NAME,PASSWD) "[lindex $argv $argi]"; incr argi
set FLAGS($NE_NAME,RMT_DIR) "[lindex $argv $argi]"; incr argi
set FLAGS($NE_NAME,FILE_SPEC) "[lindex $argv $argi]"; incr argi
set FLAGS($NE_NAME,POLL_INT) "[lindex $argv $argi]"; incr argi
set GROUP_MEMBERS $NE_NAME
set GROUP_COUNT 1
#
# If this a grouped ftp_pull process then query the database for all
the
# grouped NE systems' parameter $NE_NAME. The NE_NAME variable holds
the
# group name in this case. If this is not a grouped ftp_pull then
simply
# query the database for the specified NE name/
#
} elseif { $GROUP_MODE } {
if { [query_database_group $NE_NAME]>0 } { return 2 }
} else {
if { [query_database_single $NE_NAME]>0 } { return 2 }
}
# Everything is OK
return 0
}
#-------------------------------------------------------------------------------
# Each call to this function returns a random number between 0 and
$max. The
# function random_init {} needs to be called at the beginning of thre
program.
#-------------------------------------------------------------------------------
proc random {max} {
global _random _period
set _random [expr ($_random*7141 + 54773) % $_period]
expr int($max*($_random/double($_period)))
}
#-------------------------------------------------------------------------------
# The function initializes the seed variables for the random {} random
number
# generator function
#-------------------------------------------------------------------------------
proc random_init {seed} {
global _random _period
set _random $seed
set _period 259200
}
#-------------------------------------------------------------------------------
# This function shuffles the order of the files in the FILES array.
#-------------------------------------------------------------------------------
proc shuffle_files {} {
global FILES
debug 1 "proc shuffle_files"
set fcnt [array size FILES]
for {set n1 0; set n2 0} {$n1<$fcnt} {incr n1} {
set n2 [expr ($n2 + [random $fcnt]) % $fcnt]
debug 3 "n1=$n1 n2=$n2"
set files $FILES($n1)
set FILES($n1) $FILES($n2)
set FILES($n2) $files
}
}
#-------------------------------------------------------------------------------
# This function provides a consistant method for sending a command to
an FTP
# expect session. After the command is sent, the functon waits for FTP
command
# prompt and any potential error conditions.
#-------------------------------------------------------------------------------
proc ftp_cmd {CMD} {
global spawn_id timeout
debug 2 "proc ftp_cmd \"$CMD\""
if { [string length $CMD]>0 } {
send "$CMD\r"
}
while {1} {
expect {
timeout {
log_msg "ERROR - ftp cmd, \"$CMD\", timed out"
set FAULT 2
break
}
eof {
log_msg "ERROR - ftp cmd, \"$CMD\", detected end-of-file"
set FAULT 3
break
}
-re "\[\r\n](\[^\r\n]+)" {
set LINE $expect_out(1,string)
set CODE [lindex $LINE 0]
debug 2 "CODE=\"$CODE\" LINE=\"$LINE\""
if { "$CODE"=="ftp>" } {
set FAULT 0
break
} elseif { [regexp "^\[45]\[0-9]\[0-9]$" "$CODE"]>0 } {
if { [string match "bytes" [lindex $LINE 1]] && [string match
"received" [lindex $LINE 2]] } { continue }
log_msg "ERROR - ftp_cmd \"$CMD\", failed ($LINE)"
set FAULT 1
continue
}
}
}
}
return $FAULT
}
#-------------------------------------------------------------------------------
# This function provides a consistant method for starting an FTP
expect session
#-------------------------------------------------------------------------------
proc ftp_open {IPADDRESS} {
global spawn_id timeout
global ftp_ip_addr
global LOGIN PASSWD
global FTP_MODE TIMEOUT
# This logic makes sure the FTp child process is open to the desired
IP address.
# If the IP address has changed the current FTP process is closed
down first.
if { [string length $ftp_ip_addr]>0 && [string match $ftp_ip_addr
$IPADDRESS] } {
ftp_close
}
if { $spawn_id != 0 } { return 0 }
debug 1 "proc ftp_open"
set timeout $TIMEOUT
spawn ftp -inv $IPADDRESS
if { [ftp_cmd ""]>0 } { ftp_close; return 1 }
if { [ftp_cmd "user $LOGIN $PASSWD"]>0 } { ftp_close; return 1}
if { $FTP_MODE=="ascii" } { if { [ftp_cmd "ascii"]>0 } { ftp_close;
return 1} }
if { $FTP_MODE=="binary" } { if { [ftp_cmd "binary"]>0 } { ftp_close;
return 1} }
return 0
}
#-------------------------------------------------------------------------------
# This function provides a consistant method for ending an FTP expect
session
#-------------------------------------------------------------------------------
proc ftp_close {} {
global spawn_id
debug 1 "proc ftp_close"
send "bye\r"
close
wait
if { $spawn_id != 0 } { set spawn_id 0 }
return 0
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc ftp_pwd {} {
global spawn_id timeout
debug 1 "proc ftp_pwd"
set rdir ""
if { $spawn_id==0 } { return $rdir }
send "pwd\r"
# scan the results of the pwd command and capture the returned
directory
while {1} {
expect {
timeout {
log_msg "ERROR - ftp cmd, \"pwd\", timed out"
set FAULT 2
break
}
eof {
log_msg "ERROR - ftp cmd, \"pwd\", detected end-of-file"
set FAULT 3
break
}
-re "\[\r\n](\[^\r\n]+)" {
set LINE $expect_out(1,string)
set CODE [lindex $LINE 0]
debug 2 "CODE=\"$CODE\" LINE=\"$LINE\""
if { "$CODE"=="ftp>" } {
set FAULT 0
break
} elseif { [regexp "^\[45]\[0-9]\[0-9]$" "$CODE"]>0 } {
log_msg "ERROR - ftp_cmd \"pwd\", failed ($LINE)"
set FAULT 1
return $rdir
continue
} elseif { [regexp "^\[2]\[0-9]\[0-9]$" "$CODE"]>0 } {
set FAULT 0
set rdir "[lindex $LINE 1]"
set rdir "[string trimright $rdir "\""]"
set rdir "[string trimleft $rdir "\""]"
continue
}
}
}
}
return $rdir
}
#-------------------------------------------------------------------------------
# This function retrieves a list of file names on the remote system.
Only file
# matching the FILE_SPEC on the command line are retrieved. Also, the
length
# in bytes of each remote file are also retrieved.
#-------------------------------------------------------------------------------
proc list_files {} {
global FILES
global IP_ADDR LOGIN PASSWD RMT_DIR FILE_SPEC
global REMOTE_OS PULL_ORDER
global spawn_id timeout
debug 1 "proc list_files"
ftp_open $IP_ADDR
if { [info exists FILES] } { unset FILES }
set rmt_home "[ftp_pwd]"
# cd into each remote directory and get a list of the file in each
set cnt 0
foreach rdir $RMT_DIR {
debug 2 "list_files - scanning directory $rdir"
ftp_cmd "cd \"$rmt_home\""
ftp_cmd "cd \"$rdir\""
if { $PULL_ORDER=="T" && ([string match "unix" $REMOTE_OS] ||
[string match "windows" $REMOTE_OS]) } {
debug 1 "send \"ls -ltr\""
send "ls -ltr\r"
} elseif { [string match "vms" $REMOTE_OS] } {
ftp_cmd "verbose"
debug 1 "send \"ls\""
send "ls\r"
} else {
debug 1 "send \"dir\""
send "dir\r"
}
# scan the results of the ls comamd and capture the complete line
# specs on the files of interest.
while {1} {
expect {
"ftp>" { break }
-re "\[\r\n](\[^\r\n]+)\[\r\n]" {
if { [string match "vms" $REMOTE_OS] } {
set file $expect_out(1,string)
set length -1
} elseif { [string match "windows" $REMOTE_OS] } {
set line $expect_out(1,string)
set length [lindex $line 2]
set file [lindex $line 3]
} else {
set line $expect_out(1,string)
set perms [lindex $line 0]
set length [lindex $line 4]
set file [lrange $line 8 [expr [llength $line]-1]]
if { [string index $perms 0] != "-" } { continue }
}
# This for loop compares the current files against each
# filename specification.
foreach spec $FILE_SPEC {
if { [string match $spec $file] } {
set FILES($cnt) "{$rdir} {$file} $length"
incr cnt
}
}
}
}
}
}
ftp_cmd "cd \"$rmt_home\""
return $cnt
}
#-------------------------------------------------------------------------------
# The function automates the retrieval of one file from the remote
system.
#-------------------------------------------------------------------------------
proc get {rmt_file lcl_file} {
global IP_ADDR
debug 1 "proc get"
# If the get file already exist locally, it was was left over from
# a prior aborted get call, so delete the local file first
if { [file exists $lcl_file]==1 } { exec rm -f $lcl_file }
ftp_open $IP_ADDR
if { [ftp_cmd "get \"$rmt_file\" \"$lcl_file\""]>0 } { ftp_close;
return 1}
return 0
}
#-------------------------------------------------------------------------------
# The function automates the deletion of one file on the remote
system.
#-------------------------------------------------------------------------------
proc delete {rmt_file} {
global IP_ADDR
debug 1 "proc delete"
ftp_open $IP_ADDR
if { [ftp_cmd "delete \"$rmt_file\""]>0 } { ftp_close; return 1}
return 0
}
#-------------------------------------------------------------------------------
# The function automates the renaming of one file on the remote
system.
#-------------------------------------------------------------------------------
proc rename {old_file new_file} {
global IP_ADDR
debug 1 "proc rename"
ftp_open $IP_ADDR
if { [ftp_cmd "rename \"$old_file\" \"$new_file\""]>0 } { ftp_close;
return 1}
return 0
}
#-------------------------------------------------------------------------------
# The function automates the moving of one file to another directory
# If the new_dir spec begins with a '/' then move file absolutely.
# If new_dir does not begin with '/' then move to new directory
relative
# to rmt_dir spec.
#-------------------------------------------------------------------------------
proc move {file old_dir new_dir} {
global IP_ADDR
debug 1 "proc move"
ftp_open $IP_ADDR
if { [string match "/*" $new_dir] } {
set target_dir "$new_dir"
} else {
set target_dir "$old_dir/$new_dir"
}
if { [ftp_cmd "rename \"$old_dir/$file\" \"$target_dir/$file\""]>0 }
{ ftp_close; return 1 }
return 0
}
#-------------------------------------------------------------------------------
# This function provides a consistant method for sending a command to
an SQLPLUS
# expect session. After the command is sent, the functon waits for sql
command
# prompt or any potential error conditions.
#-------------------------------------------------------------------------------
proc sql_cmd {CMD} {
global spawn_id timeout
global SQL_RESULTS SQL_CNT
debug 1 "proc sql_cmd \"$CMD\""
if { [string length $CMD]>0 } {
send "$CMD\r"
}
set FAULT 0
set SQL_CNT 0
set SQL_RESULTS($SQL_CNT) ""
while {1} {
expect {
timeout {
log_msg "ERROR - sql cmd, \"$CMD\", timed out"
set FAULT 2
break
}
eof {
log_msg "ERROR - sql cmd, \"$CMD\", detected end-of-file"
set FAULT 3
break
}
-re "\[\r\n](\[^\r\n]+)" {
set LINE $expect_out(1,string)
set CODE [lindex $LINE 0]
debug 2 "CODE=\"$CODE\" LINE=\"$LINE\""
if { "$CODE"=="SQL>" } {
break
} elseif { [regexp "^\[A-Z]\{3\}-\[0-9]\{5\}:$" "$CODE"]>0 } {
log_msg "ERROR - sql_cmd ($LINE)"
set FAULT 1
continue
} else {
set SQL_RESULTS($SQL_CNT) "$LINE"
incr SQL_CNT
}
}
}
}
return $FAULT
}
#-------------------------------------------------------------------------------
# This function provides a consistant method for starting an SQLPLUS
session
#-------------------------------------------------------------------------------
proc sql_open {} {
global spawn_id timeout oracle_user oracle_passwd
global TIMEOUT
if { $spawn_id != 0 } { return 0 }
debug 1 "proc sql_open"
set timeout $TIMEOUT
spawn sqlplus $oracle_user/$oracle_passwd
if { [sql_cmd ""]>0 } { sql_close; return 1 }
if { [sql_cmd "set feedback OFF"]>0 } { sql_close; return 1 }
if { [sql_cmd "set heading OFF"]>0 } { sql_close; return 1 }
if { [sql_cmd "set linesize 499"]>0 } { sql_close; return 1 }
return 0
}
#-------------------------------------------------------------------------------
# This function provides a consistant method for ending an SLQPLUS
session
#-------------------------------------------------------------------------------
proc sql_close {} {
global spawn_id
debug 1 "proc sql_close"
send "exit\r"
close
wait
if { $spawn_id != 0 } { set spawn_id 0 }
return 0
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc query_database { ne_name } {
global FLAGS
global SQL_RESULTS SQL_CNT
debug 1 "proc query_database"
if { [sql_cmd "select
NE_NAME,NE_IP,NE_LOGIN,NE_PASSWORD,NE_POLL_INTERVAL from NE_FTP_Config
where NE_NAME = '$ne_name';"]>0 } { return 1}
if { $SQL_CNT < 1 } {
log_msg "NE_NAME, $ne_name, not found in NE_FTP_Config table"
return 1
}
debug 2 "SQL_RESULTS = \"$SQL_RESULTS(0)\""
if { [llength $SQL_RESULTS(0)]<5 } {
log_msg "Incomplete data found for NE_NAME, $ne_name"
return 1
}
set ip_addr [lindex $SQL_RESULTS(0) 1]
set ip_addr [string trimright $ip_addr ":"]
set FLAGS($ne_name,IP_ADDR) $ip_addr
set FLAGS($ne_name,LOGIN) "[lindex $SQL_RESULTS(0) 2]"
set FLAGS($ne_name,PASSWD) "[lindex $SQL_RESULTS(0) 3]"
set FLAGS($ne_name,POLL_INT) [lindex $SQL_RESULTS(0) 4]
if { [sql_cmd "select NE_REMOTE_DIR from NE_FTP_Config where NE_NAME
= '$ne_name';"]>0 } { return 1}
debug 2 "SQL_RESULTS = \"$SQL_RESULTS(0)\""
if { [string length $SQL_RESULTS(0)]<1 } {
log_msg "Incomplete data found for NE_NAME, $ne_name"
return 1
}
set FLAGS($ne_name,RMT_DIR) "$SQL_RESULTS(0)"
if { [sql_cmd "select NE_FILE_SPEC from NE_FTP_Config where NE_NAME =
'$ne_name';"]>0 } { return 1}
debug 2 "SQL_RESULTS = \"$SQL_RESULTS(0)\""
if { [string length $SQL_RESULTS(0)]<1 } {
log_msg "Incomplete data found for NE_NAME, $ne_name"
return 1
}
set FLAGS($ne_name,FILE_SPEC) "$SQL_RESULTS(0)"
if { [sql_cmd "select NE_OPTIONS from NE_FTP_Config where NE_NAME =
'$ne_name';"]>0 } { return 1}
debug 2 "SQL_RESULTS = \"$SQL_RESULTS(0)\""
# Capture the retrieved results
if { [string length $SQL_RESULTS(0)]>0 } {
set ne_options "$SQL_RESULTS(0)"
parseflags $ne_name "$ne_options"
}
return 0
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc query_database_single { ne_name } {
global SQL_RESULTS SQL_CNT
global GROUP_MEMBERS GROUP_COUNT
debug 1 "proc query_database_single"
sql_open
set GROUP_MEMBERS ""
set status "[query_database $ne_name]"
lappend GROUP_MEMBERS $ne_name
set GROUP_COUNT 1
sql_close
return $status
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc query_database_group { grp_name } {
global SQL_RESULTS SQL_CNT
global GROUP_MEMBERS GROUP_COUNT
debug 1 "proc query_database_group"
sql_open
if { [sql_cmd "select NE_NAME from NE_GROUP where
GROUP_NAME='$grp_name';"]>0 } {
sql_close
return 1
}
if { $SQL_CNT < 1 } {
log_msg "GROUP_NAME, $grp_name, not found in NE_GROUP table"
sql_close
return 1
}
set GROUP_MEMBERS ""
set cnt $SQL_CNT
for {set GROUP_COUNT 0} {$GROUP_COUNT<$cnt} {incr GROUP_COUNT} {
set ne_name $SQL_RESULTS($GROUP_COUNT)
if { [query_database $ne_name] > 0 } {
log_msg "Group query failed before completion, NE_NAME=$ne_name"
sql_close
return 1
}
lappend GROUP_MEMBERS $ne_name
}
sql_close
return 0
}
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
proc signal_handler {} {
global OK2RUN SLEEPING
log_msg "Signal SIG[trap -name] caught. exiting..."
set OK2RUN 0
if { $SLEEPING } { exit 0 }
}
#-------------------------------------------------------------------------------
# main thread
#-------------------------------------------------------------------------------
trap signal_handler { SIGHUP SIGINT SIGQUIT SIGTERM }
# Whenever the OK2RUN variable get set to 0 the script will gracefully
stop
set OK2RUN 1
# This variable help shutdown the ftp_pull script when a signal is
caught. If
# the script is sleeping at the moment a signal is caught this
variable
# indicates such to the signal handler, which then immediately ewxits
the script.
set SLEEPING 0
# stop the underlying dialogue text from echoing to standard output
log_user 0
# initialize the spawn_id variable to indicate that no child processes
are running
set spawn_id 0
set LOGFILE ""
set THEN ""
set TIMEOUT 300
set DEBUG 0
log_user 0
exp_internal 0
set GROUP_MODE 0
set GROUP_MEMBERS ""
set GROUP_COUNT 0
set LOCALHOST [exec uname -n]
set CCD_DATA $env(CCD_DATA)
set LOGDIR $env(LOGDIR)
set ftp_ip_addr ""
set empty_count 0
set oracle_user $env(ORACLE_USER)
set oracle_passwd $env(ORACLE_PASSWD)
set FLAGS ""
initflags
if { [parsecommandline] > 0 } { send_user "$USAGE\n"; exit 2 }
setflags $NE_NAME
if { $LOGDIR=="" } { set LOGDIR "/tmp" }
debug 1 "main thread"
# Initialize the random number generator
random_init 123
while { $OK2RUN } {
debug 3 "GROUP_MEMBERS=$GROUP_MEMBERS"
foreach NE_NAME $GROUP_MEMBERS {
setflags $NE_NAME
set POLL_INT [expr $POLL_INT*60]
set DATADIR $CCD_DATA/nehd/$NE_NAME
# Now moving files directly to Transferred directory
#set CREATDIR $DATADIR/Creating
set TRANSDIR $DATADIR/Transferred
# Make sure that the local Creating and Transferred directories
# have been created
# Now moving files directly to Transferred directory
#if { [file isdirectory ${CREATDIR}]==0 } {
#log_msg "Directory, $CREATDIR , does not exist"
#exit 1
#}
if { [file isdirectory ${TRANSDIR}]==0 } {
log_msg "Directory, $TRANSDIR , does not exist"
exit 1
}
# First open or reopen the FTP session to the remote system
ftp_open $IP_ADDR
if { !$OK2RUN } break
#
# Get a list of files to be transferred on the remote system. If
there are no
# new remote files, then increment the empty_count value by one. When
the empty
# count equals or exceeds the number of NE systems in the group, then
all
# the NE systems have no files ready to pull. In this case sleep for
a while.
#
set fcnt [list_files]
if { $fcnt==0 && $empty_count<$GROUP_COUNT } { incr empty_count }
debug 3 "fcnt=$fcnt empty_count=$empty_count GROUP_COUNT=
$GROUP_COUNT"
# This conditional tests to see if all the remote NEs have no files.
# If so reset the empty_count value and then sleep for a while.
if { $empty_count>=$GROUP_COUNT } {
ftp_close
set empty_count 0
log_msg "No files ready for retrieval from $NE_NAME remote system"
log_msg "No files ready for retrieval from remote system(s),
sleeping..."
set SLEEPING 1
sleep $POLL_INT
set SLEEPING 0
continue
# If the current remote NE system has no files ready for retrieval,
then
# continue on to the next remote NE system for polling.
} elseif { $fcnt==0 } {
ftp_close
log_msg "No files ready for retrieval from $NE_NAME remote system"
continue
# If the current NE system has files ready for retrieval then reset
the
# empty count back to zero.
} else {
set empty_count 0
}
if { !$OK2RUN } break
# If the shuffle option (-s) is used then shuffle the order of
# the file in the FILES arrays.
if { $PULL_ORDER=="R" } { shuffle_files }
# For each remote file do the following 4 steps
set cnt -1
while { $OK2RUN && [incr cnt]<$fcnt } {
# If the maxpulls option (-M) was used then test whether the
# maximum number of files per session. If so then flag the NE
# system as empty and proceed to polling the next NE system.
if { $cnt>=$MAXPULLS } {
ftp_close
log_msg "The maximum number of files were pulled from $NE_NAME"
incr empty_count
break
}
set rmt_dir "[lindex $FILES($cnt) 0]"
set rmt_file "[lindex $FILES($cnt) 1]"
set rmt_size [lindex $FILES($cnt) 2]
# The ';' '\s\' and '\t' chars found in some remote systems files
will
# cause problems with UNIX files so replace them with a '_'
character
regsub "\[ ;]" $rmt_file "_" lcl_file
if { $PREPEND_NENAME==1 } {
set lcl_file "$NE_NAME.$lcl_file"
}
# Now moving directly to Transferred Directory.
if { [get $rmt_dir/$rmt_file $TRANSDIR/$lcl_file.tmp]>0 } {
continue
}
debug 2 "ftp GET command succeeded"
# Now moving directly to Transferred Directory.
if { [file exists "$TRANSDIR/$lcl_file.tmp"]==0 } {
log_msg "ERROR - ftp GET failed, local copy of file missing"
continue
}
# Determine the size in bytes of the created local file
# Now moving directly to Transferred Directory.
#set lcl_size [file size $CREATDIR/$lcl_file]
set lcl_size [file size $TRANSDIR/$lcl_file.tmp]
debug 2 "lcl_size=$lcl_size rmt_size=$rmt_size"
# Now compare the size of the local file with the reported size
# of the file on the remote system. If they do not match, delete
# the partial local copy of the file.
if { $rmt_size < 0 } {
log_msg "WARNING - remote file length info, skipping local/remote
comparison"
} elseif {$lcl_size!=$rmt_size} {
# Now moving directly to Transferred Directory.
#exec rm -f $CREATDIR/$lcl_file
exec rm -f $TRANSDIR/$lcl_file.tmp
log_msg "ERROR - $lcl_file.tmp local/remote file sizes mismatch,
$lcl_size/$rmt_size \([expr $cnt+1] of $fcnt\)"
continue
} else {
debug 2 "Local file size matches remote file size"
}
# FTP DELETE the file on the remote system
if { $FILE_DISPO=="d" } {
if { [delete $rmt_dir/$rmt_file]!=0 } { continue }
debug 2 "Remote file deleted"
} elseif { $FILE_DISPO=="m" } {
if { [move $rmt_file $rmt_dir $MOVEDIR]!=0 } { continue }
debug 2 "Remote file moved"
} elseif { $FILE_DISPO=="r" } {
set new_file "[exec echo $rmt_file | sed $RENAMESPEC]"
if { [rename $rmt_dir/$rmt_file $rmt_dir/$new_file]!=0 }
{ continue }
debug 2 "Remote file renamed"
}
# Now that the file has been sucessfully retrieved from the remote
# system, move it to the Transferred directory
# No need this since file has been moved to Transfered directory
already
# howeverver rename the file name to not have .tmp extension
#exec mv -f $CREATDIR/$lcl_file $TRANSDIR/$lcl_file
exec mv -f $TRANSDIR/$lcl_file.tmp $TRANSDIR/$lcl_file
# Write a message in the local log file that transfer was completed
log_msg "\([expr $cnt+1] of $fcnt\) Remote File $IP_ADDR:$rmt_dir/
$rmt_file\n Pulled To $NE_NAME/Transferred/
$lcl_file\n"
}
}
}
ftp_close
exit 0
.
- Follow-Ups:
- Re: Expect - memory leak issue
- From: Cameron Laird
- Re: Expect - memory leak issue
- References:
- Expect - memory leak issue
- From: JD
- Re: Expect - memory leak issue
- From: Gerald W. Lester
- Re: Expect - memory leak issue
- From: Cameron Laird
- Expect - memory leak issue
- Prev by Date: Re: Pointing to working directory
- Next by Date: Re: Pointing to working directory
- Previous by thread: Re: Expect - memory leak issue
- Next by thread: Re: Expect - memory leak issue
- Index(es):
Relevant Pages
|
