Re: Why would it appear to my scripts that a server they're connecting to has gone away (using Net::FTP and LWP)
- From: Ted Byers <r.ted.byers@xxxxxxxxx>
- Date: Mon, 8 Dec 2008 12:22:00 -0800 (PST)
On Dec 6, 5:57 pm, Bill H <b...@xxxxxxxxx> wrote:
On Dec 5, 1:23 am, Ted Byers <r.ted.by...@xxxxxxxxx> wrote:
Here is the interesting part of the log for my script that uses
NET::FTP:
Copying 44 files
Net::FTP=GLOB(0x3e18a04)>>> TYPE I
Net::FTP=GLOB(0x3e18a04)<<< 200 Type set to I
Status: 2
C_MerchantData_ALT_Work.2008-11-12.zip
Net::FTP=GLOB(0x3e18a04)>>> ALLO 5062889
Net::FTP=GLOB(0x3e18a04)<<< 202 No storage allocation neccessary.
Net::FTP=GLOB(0x3e18a04)>>> PORT 10,1,10,124,7,5
Net::FTP=GLOB(0x3e18a04)<<< 200 Port command successful
Net::FTP=GLOB(0x3e18a04)>>> STOR C_MerchantData_ALT_Work.
2008-11-12.zip
Net::FTP=GLOB(0x3e18a04)<<< 150 Opening data channel for file
transfer.
Net::FTP=GLOB(0x3e18a04): Timeout at C:/Perl/site/lib/Net/FTP/
dataconn.pm line 7
4
Status: 1
C_MerchantData_ALT_Work.2008-11-13.zip
Net::FTP=GLOB(0x3e18a04)>>> ALLO 5299854
Net::FTP: Unexpected EOF on command channel at C:\backup
\copy.backups.to.T.O.pl
line 48
Status: 5
Qualitatively, it looks like only one zip archive is tranfered (and my
colleagues in TO can open them and verify the contents), and then the
ftp session seems to go away and there is no more interaction with the
ftp server.
Here is the interesting part of a log created (LWP::DebugFile):
Here is the last bit pertaining to the download that occured just
before the failed dowwnload:
LWP::Protocol::http::request: ()
# Time now: {1228452248} = Thu Dec 4 23:44:08 2008
LWP::UserAgent::request: Simple response: OK
# LWP::DebugFile logging to lwp.log
# Time now: {1228452249} = Thu Dec 4 23:44:09 2008
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
ignoring the URL requested (it doesn't matter because if I select a
temporal interval involving a few kbytes of data, all works fine),
here is what follows:
.LWP::Protocol::http::request: ()
# Time now: {1228452869} = Thu Dec 4 23:54:29 2008
LWP::UserAgent::request: Simple response: Internal Server Error
# LWP::DebugFile logging to lwp.log
# Time now: {1228452870} = Thu Dec 4 23:54:30 2008
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
I find it interesting to observe that simply including use of
LWP::DebugFile is sufficient to prevent the script from freezing or
locking up. Instead, an error like this is logged and the script
proceeds to the next request. The error shown above happens in the
middle of the download, and in this run, this is the only download
that failed.
Questions:
1) How do I get more information from LWP::DebugFile? Or is there any
more information to be had?
2) I'd also included the following statements:
use Log::Trace;
use Log::Trace 'print';
use Log::Trace log => 'master.generic.download.trace.log';
use Log::Trace warn => { Deep => 1, Everywhere => 1, Verbose => 2 };
But I did not get any log from Log::Trace. i must have missed
something, but what?
3) The information I'm getting from Net::FTP seems fine, except I have
yet to find the documentation for what the status codes (returned by
the status function) mean. (I found curl, but haven't had time to
compile and run it on the same files). Where will I find a more
complete description of these status codes?
4) Most importantly of all, why do these transfers fail and what can
be done about them?
Thanks
Ted
Ted, not sure if this may be your problem but I had an issue with
NET::Ftp earlier this year. I have a script that runs as a cron job
and uses ftp to transfer files from a clients web server to their
interanl server. It had always worked great, then we upgraded the
server and it would never connect and just get timeout warnings, even
though I could connect from my own machine perfectly fine. It took me
a day or so to determine that the upgraded server was not allowing the
outgoing ftp connection because the destination server was an IP
address. After I setup a DNS for it and used a domain name in the
connection, it started working just fine again.
Bill H- Hide quoted text -
- Show quoted text -
Thanks all.
We're on our way to a solution, but not quite there, yet. My
colleague (actually the ftp/router/network administrator) is convinced
it is largely a timeout error. He retried my script with a pile of
really small files (a few k each) and my script worked flawlessly.
The files that we need to copy to our head office, though, are
typically a few megabytes in size, with one or two that are larger
(e.g. a dump of the current state of a DB managed using MySQL, which
is 200 MB before being compressed and about 15 MB after being
compressed).
I guess, then, the new infomrmation here is that this problem exists
only for transfers of large files, and that all works correctly if we
process only tiny files.
But then, this is a similar problem to our script that uses LWP to
retrieve data from our data feed, in that if the amount of data is
small, all is fine, but there is a server error in the client log when
LWP is asked to retrieve a few MB from the same server (in the LWP
case, we don't have direct access to the server, so we're not sure if
the problem is our script, LWP or the other guy's server).
Did I mention I am paranoid about backing up our data and code? ;-)
Does anyone else use Net::FTP or LWP's user agents to move around
megabytes of data, in a single file? If so, are there any tricks of
the trade you'd care to share?
Thanks
Ted
.
- Follow-Ups:
- References:
- Prev by Date: Re: is perl obsolete?
- Next by Date: Re: is perl obsolete?
- Previous by thread: Re: Why would it appear to my scripts that a server they're connecting to has gone away (using Net::FTP and LWP)
- Next by thread: Re: Why would it appear to my scripts that a server they're connecting to has gone away (using Net::FTP and LWP)
- Index(es):
Relevant Pages
|
