Re: LWP module - parse one line at a time (only download part of a page)
- From: xhoster@xxxxxxxxx
- Date: 21 Jan 2006 00:16:39 GMT
"nobull@xxxxxxxx" <nobull67@xxxxxxxxx> wrote:
> Alf McLaughlin wrote:
>
> > I want to download a fairly large amount of data from a webpage
> > (~10MB), but the stuff I'm really interested in is always toward the
> > top of the page (however, I don't know exactly where). Since I'm only
> > interested in two or three lines, I don't want to download the whole
> > page. I would like download until I see what I want (such as my
> > $current_line =~ /WHAT I WANT/) and then kill the download.
>
> Read the description of the get() method of LWP::UserAgent.
I think you mean request() rather than get().
> In particular note the existance of the callback and the bit where it
> says "The callback can abort the request by invoking die()."
This method is the direct answer to the OPs question, but he will have to
be careful to account for the chance that his desired string will span a
chunk boundary.
I think a simpler but less rigorous option would be to set the
$ua->max_size to his best guess of a upper limit on how far into the
response the desired string can be. But there is always the danger that
the upper limit turns out to be set too low, and you miss things that the
callback method would find. Of course, there is the corresponding hazard
that the guess will be set too high, and he will still be reading far more
data than necessary.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
.
- References:
- LWP module - parse one line at a time (only download part of a page)
- From: Alf McLaughlin
- Re: LWP module - parse one line at a time (only download part of a page)
- From: nobull@xxxxxxxx
- LWP module - parse one line at a time (only download part of a page)
- Prev by Date: Re: LWP module - parse one line at a time (only download part of a page)
- Next by Date: Re: Cant load ...BerkeleyDB.so for module BerkeleyDB: libdb-4.4.so
- Previous by thread: Re: LWP module - parse one line at a time (only download part of a page)
- Next by thread: Re: LWP module - parse one line at a time (only download part of a page)
- Index(es):
Relevant Pages
|
