LWP cookies
From: Richard Bell (rbell01824_at_comcast.net)
Date: 03/28/04
- Next message: John Bokma: "Re: LWP cookies"
- Previous message: Richard Bell: "Re: problem/help with using LWP"
- Next in thread: John Bokma: "Re: LWP cookies"
- Reply: John Bokma: "Re: LWP cookies"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Sun, 28 Mar 2004 17:41:46 GMT
I'm using LWP to scrape some data from a DB over the web. Eventually I
end up executing the following code:
#!/usr/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
<much code deleted>
### request number 18 ###
my $req18 = HTTP::Request->new(POST => '
http://www2.fdic.gov/sdi/download_exect.asp ');
$req18->header('Host' => 'www2.fdic.gov');
$req18->header('User-Agent' => 'Mozilla/5.0 (X11; U; Linux i686; en-US;
rv:1.4.1) Gecko/20031030');
$req18->header('Accept' =>
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1');
$req18->header('Accept-Language' => 'en-us,en;q=0.5');
$req18->header('Accept-Encoding' => 'gzip,deflate');
$req18->header('Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7');
$req18->header('Keep-Alive' => '300');
$req18->header('Proxy-Connection' => 'keep-alive');
$req18->header('Referer' =>
'http://www2.fdic.gov/sdi/download_FindInsts.asp');
$req18->header('Cookie' => 'CustRepUserID=; InstSelected=Yes;
HasCustReps=Yes; RepSelChanged=No; ActiveRepName=;
rep3x=9000=&8000=&7000=&6000=&5000=&99=namehcr%2Coffdom%2Cofffor%2Cstmult%2Cwebaddr%2C&30=nimy%2Croa%2Croe%2Ceeffr%2Castempm%2C&4000=&3000=;
rep1x=16000=&12100=&16=&15=&16300=&14=dep%2Cdepfor%2Cdepdom%2Ctrn%2Cddt%2Cdepsmamt%2Cdepsmb%2Cdeplgamt%2Cdeplgb%2C&16100=&12200=&1010=&13=&12000=&12=&11000=&11=&1000=&16200=&12300=&10000=&1110=&10=numemp%2Casset%2C;
rep2x=23=&26000=&27500=&24000=&24=&27100=&22=&27400=&22000=&21=&27200=&21000=&25=&27300=&25000=&20=&27000=&20000=&2000=;
CPG=HASCPG=No&CPGUSERID=64EEB2F8EDBEFB37;
rep0=USERREP7=B%26As&USERREP6=pdpa&USERREP5=EPGSales02&USERREP4=aa&USERREP3=bh10%2D28report&USERREP2=Test+MS+Security+Patches&USERREP8=BOCCPeers&USERREP1=random;
SetCookie=OK; ASPSESSIONIDSDADQBAB=BALONNDDJHGCHFNNEELFLCMD');
$req18->header('Content-Type' => 'application/x-www-form-urlencoded');
$req18->header('Content-Length' => '384');
$req18->content('SQL=SELECT+stru.Cert+from+risstru+stru++where+stru.Active+%3D+1+++++++++++and+stru.repdte+%3D+%276%2F30%2F2003%27++order+by+stru.NAME+&inFlags=&selections=10%2Cnumemp%2Casset%2C14%2Cdep%2Cdepfor%2Cdepdom%2Ctrn%2Cddt%2Cdepsmamt%2Cdepsmb%2Cdeplgamt%2Cdeplgb%2C30%2Cnimy%2Croa%2Croe%2Ceeffr%2Castempm%2C99%2Cnamehcr%2Coffdom%2Cofffor%2Cstmult%2Cwebaddr&Period1=6%2F30%2F2003&IncomeBasis=');
print "<-------------------------------------------->";
print $ua->request($req18)->as_string;
Amongst other things, this code defines a number of headers including
one for cookies. I'm unclear on several things and could use some help
from someone who actually knows how LWP does its magic with regard to
cookes.
I assume the form is not posted until the line
print $ua->request($req18)->as_string;
executes. Is this correct?
The cookies header contains a bunch of stuff, including a "session
cookie" that expires. While I'm not entirely sure, I think it's this bit:
ASPSESSIONIDSDADQBAB=BALONNDDJHGCHFNNEELFLCMD
That is the cookie and value when this code was originally got (some
days ago) using wsp.pl and template.pl. But the cookie, of course,
expires. If I revisit the page that issues the cookie (a new value for
ASPSESSIONIDSDADQBAB) will the new value be sent along as part of the
'request' or do I need to modify the Cookie header? Does it 'override'
the value specified in the header?
This is causing me considerable confusion as when I run a perl script
that goes through all 18 pages/forms based on wsp and template run some
days ago (with a now expired session cookie), I'm able to fetch the data
OK. Clearly some current correct session cookie is being sent and all
is well. Unfortunately, when I run a perl script that only visits the
page that seems to be issuing ASPSESSIONIDSDADQBAB and then run the
above code (just $req18 stuff), I get a return page that indicates that
the session has expired. Any clues?
Thanks
R
- Next message: John Bokma: "Re: LWP cookies"
- Previous message: Richard Bell: "Re: problem/help with using LWP"
- Next in thread: John Bokma: "Re: LWP cookies"
- Reply: John Bokma: "Re: LWP cookies"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
