Re: why getstore can't work, especially for store xml file on freebsd

I will do further research according to your suggestions. Thank you very much!

On 5/26/05, Jay Savage <daggerquill@xxxxxxxxx> wrote:
> On 5/24/05, Franklin <zhgzhang@xxxxxxxxx> wrote:
> > Hi;
> >
> > I have a small program that uses getstore to fetch a xml webpahe and
> > store it. It is:
> >
> > use strict;
> > use warnings;
> >
> > use LWP::Simple;
> >
> > my $xmlurl="http://www.sec.gov/Archives/edgar/data/903404/000118143105029692/rrd79736.xml";;
> > my $xmlfile="xml.xml";
> > my $status=getstore($xmlurl,$xmlfile);
> >
> > if(is_success($status))
> > {
> > print "xmlfile store is ok\n";
> > }
> > else
> > {
> > print "xmlfile store isn't ok\n";
> > }
> >
> >
> > It works well on windows platform, but doesn't on freebsd. That is,
> > every time the getstore function's response is "200 ok", but the
> > stored xml file isn't correct, it means, it isn't as same as the xml
> > webpage. What is wrong with it? And is there any solution or
> > alternatives to this problems?
> >
> > Thank you very much!
> >
> > Best wishes
> > Franklin
> >
> > T
>
> This sounds like a server issue; most people see this frequently when
> trying to grab pages, particularly from IIS servers, even with normal
> web browsers ("Page doesn't contain any data"). Actually, I'd say if
> you're only seeing "several" connection errors in 1,000, you're doing
> pretty well for a web app. Of course, it depends on how many is
> "several". If it's three, for instance, that's a 99.7% success rate.
> Have you tried it a thousand times on Windows?
>
> Next question: what does the code look like? Are the requests being
> sent in parallel or sequentially? Either way, the FreeBSD networking
> stack is much more efficient, and you may just be overwhelming the
> server with 1,000 requests in a way you aren't when issueing the same
> request from Windows (which Windows?). If it's your server, up the
> connection limit and see if that helps.
>
> If you really think it's FreeBSD, make sure you're not running out of
> mbufs, or memory for your tate tables 9assuming you're keeping state).
> I doubt it's FreeBSD, though.
>
> In any case, it's almost certainly not an LWP issue.
>
> One last thought, though, related to whether you're running these
> requests in sequence or in parallel: You're not using any kind of
> file locking. Nor are you performing any kind of check to see if the
> system was actually able to open the file for writing. And you're
> calling all the files xml.xml. With buffered writes (and even
> without) that has the makings race conditions and other wierdness.
> since you're (hopefully) creating all of these files within a short
> amount of time, try using File::Temp or a similar method to give each
> file a separate name. you can unlink the files once you've checked
> them against -s.
>
> HTH,
>
> -- jay
> --------------------
> daggerquill [at] gmail [dot] com
> http://www.engatiki.org
>
.

Relevant Pages