Re: web crawling program

raven wrote:
Hi,

I am trying to make 1000 searches from a site from a keyword file. I
want to automate these searches. I have copied the search form of
site and modified the post part of url so i can make necessary
modifications for the automation.

// structure of form
<form post "http://www.XXXXXXXX.com/processform.php";>
select box>// a selectbox with 100 options
input box
</form>

Program will be as follows

While not end of file
{
1st step: Read a keyword from the file and assign it $value
For each option $op on the form
//Fill form
2nd step: $op is selected
3rd step: enter $value to inputbox
4: submit

5: save the result to a file, $result
6: parse $result and save to database
}
There is no problem until step 5.When i fill and submit the
form a page is opened from the remote host and contains the table

//table

data1-- data2 -- data3

data1 and data2 is text and data 3 is a link

and i want to save data1 data2 and data3link to a database.

but when the control is gone to formprocessor on the remote host i
have no idea how to complete step 5. I mean how can i gain control and
save the result to a file and parse it.

Also this is the way i can to think of but necesserly the feasible
solution. If you can think of other options, i am all ears :)

Thank you very much for your kind response.





You will need to use CURL or similar to submit the form so you can get the information back.

And BTW - do you have permission to do this? If I saw someone doing this on one of my sites, they'd be blocked immediately - if not sooner.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@xxxxxxxxxxxxx
==================

.