Re: Web programming: issues with large amounts og data
- From: Ted Byers <r.ted.byers@xxxxxxxxx>
- Date: Wed, 3 Dec 2008 23:33:43 -0800 (PST)
On Dec 4, 12:09 am, Tim Greer <t...@xxxxxxxxxxxxx> wrote:
Ted Byers wrote:
The problem manifests itself as the download terminating and the
script appearing to freeze.
There can be a number of reasons for this. Please post the relevant
portions of the (Perl) code. This might not be a perl code issue.
--
Tim Greer, CEO/Founder/CTO, BurlyHost.com, Inc.
Shared Hosting, Reseller Hosting, Dedicated & Semi-Dedicated servers
and Custom Hosting. 24/7 support, 30 day guarantee, secure servers.
Industry's most experienced staff! -- Web Hosting With Muscle!
Hi Tim,
Thanks.
Here is a simple program that shows this problem:
use strict;
use XML::Twig;
use DBI;
use IO::File;
use POSIX qw(strftime);
use Text::ParseWords;
use LWP::RobotUA;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Request::Common;
use HTTP::Response;
use Date::Manip;
Date_Init("TZ=EST5EDT");
my $affiliate;
my $merchant;
my $uid;
my $pwd;
my $opfile;
my $db='test';
my $hostname = 'localhost';
my $port = '3306';
my $user = 'xxxxxxx';
my $dbpwd = 'yyyyyyyyyyy';
my $dbh = DBI->connect("DBI:mysql:database=$db;host=$hostname",
$user, $dbpwd, {RaiseError => 1});
my $uri = "https://our.datafeed.site/query.php";;
my $req_url;
#my $ua = LWP::UserAgent->new;
my $now_string = strftime "%a %b %d %Y Generic", localtime;
mkdir "$now_string";
chdir "$now_string";
my $start_date = $ARGV[0];
my $end_date = $ARGV[1];
my $query = "SELECT merchant_name, uid, pwd FROM merchants WHERE
is_active = 1";
my $sth = $dbh->prepare($query) or die $dbh->errstr;
$sth->execute();
while (my $ref = $sth->fetchrow_hashref()) {
$merchant = $ref->{'merchant_name'};
$uid = $ref->{'uid'};
$pwd = $ref->{'pwd'};
$req_url = "$uri?username=$uid&password=$pwd&start_date=
$start_date&end_date=$end_date";
STDOUT->print($req_url);STDOUT->print("\n\n");
$opfile = "$uid.xml";
STDOUT->print($opfile);STDOUT->print("\n");
system("..\\generic_download_child.pl $uid \"$req_url\" 1>
$uid.stdout 2>$uid.stderr");
# open(MYOUTPUT,"> $opfile");
# my $response = $ua->request(POST "$uri", ['username' => "$uid",
'password' => "$pwd",'start_date' => "$start_date",'end_date' =>
"$end_date"]);
# if ($response->is_success) {
# print MYOUTPUT $response->content;
# print $response->content;
# } else {
# print STDERR $response->status_line, "\n";
# }
}
chdir "..";
Note, the string works for short time periods and a handful of
merchants is the presently commented out code is uncommented and the
call to generic_download_child.pl is commented out. But the only way
I could make this work for one week's worth of data was to move the
code that is commented out here into generic_download_child.pl . The
script shown here, when used as a driver or master for
generic_download_child.pl works fine for one week's worth of data,
but some merchants have so much data that even
generic_download_child.pl freezes if I try to get two weeks' worth of
data at a time.
What would you recommend I do to prevent this script from freeezing
while still allowing retrieval of larger chuncks of data?
In general, what would you add to this sort of script to detect any
kind of failure and take appropriate remedial action should a problem
arise? Sometimes the company that is hosting our machine has problems
with their ISP that prevents our machine from connecting to anything,
and there are occassional power issues: thus we are wrestling with
finding ways of reliably automatically detecting problems and redoing
downloads that have been interrupted. What would you be recommending
as best practice? this is a bit new to me as my strengths in software
engineering are more in the realm of high performance number crunching
and mathematical/statistical analysis (I recently completed a robust
empirical model of operational risk applicable to the services we
provide to our clients and used to manage this risk so that no one
client's problems can seriously damage us).
Thanks
Ted
.
- Follow-Ups:
- Re: Web programming: issues with large amounts og data
- From: sln
- Re: Web programming: issues with large amounts og data
- From: Tad J McClellan
- Re: Web programming: issues with large amounts og data
- References:
- Web programming: issues with large amounts og data
- From: Ted Byers
- Re: Web programming: issues with large amounts og data
- From: Tim Greer
- Web programming: issues with large amounts og data
- Prev by Date: Re: how detect english subject and predicate in a sentence
- Next by Date: FAQ 8.42 How do I make a system() exit on control-C?
- Previous by thread: Re: Web programming: issues with large amounts og data
- Next by thread: Re: Web programming: issues with large amounts og data
- Index(es):
Relevant Pages
|
