Re: How to write something to a html textfield and send it?

From: Alexander (alexhanh_at_ranssi.paivola.net)
Date: 03/13/04 

Date: 13 Mar 2004 14:11:42 -0800

+ First of all, thank you for very generous answer.

> > I have a standard html web-page with a textfield and a send button
> > added to it (look below for html). I need a way to open the page, find
> > the specific textfield, write something to it and "click" send. I need
> > to do this through a program.
>
> Do you actually need to control a particular browser and make it
> perform these actions, or do you need to write an HTTP user agent
> which can submit the form with the text field populated?

No need for controlling any particular browser. I just need to log on
first
and then start submitting data. I'm not familiar with HTTP user
agents, but
this sounds like something correct.

> And do you need a general solution for this problem, or is it only
> for this one form? And if the latter, how confident are you that
> the form won't change significantly during the time you need this
> application to run?

For this one type, and this form is very static: it will always
contain atleast these two things, the text field and the button. So
I'm quite confident.

> > I know that perl has some nice socket/http functions, which can be
> > used easily to detect the html tags and the required textfield. But
> > what about sending some data to the textfield and clicking the button?
>
> Again, the question is whether you need the browser at all. Do you
> need to write to the text field control in the browser and click the
> button, or do you just need to submit the request with the appropriate
> text?

I don't know if I need the browser for the solution. If I can "write"
into the textfield without clicking on it and typing to it through a
browser, then I don't need the browser at all. Google newsgroup is a
very close to my problem. First, I need to log on. Then I need to go
to comp.programming and click submit a new post. And then write
something to the message field, and click post message - no preview. I
need to run this process unlimited times. And the text written to the
message field is gotten from outside or generated randomly.

 
> > And no, I'm not able to do it through calling a prefixed www-address,
> > like http://www.google.com/search?q=problem to search for "problem".
>
> Most non-trivial HTML forms use HTTP POST requests. The URL in your
> example contains a query-string, which is produced by a user agent
> for a GET request.
>
> They're rather different mechanisms; with POST, the request parameters
> are passed as the HTTP content-body of the request message, in URL-
> encoded form. You take the parameter names and their values, URL-
> encode them, and put them after the HTTP header and separator in the
> request.
>
> Here's a quick example. Say there's a page with a form like
>
> <FORM action="cgi/process" method="POST">
> <P>
> Customer Name: <INPUT type="text" name="cust">
> <INPUT type="submit" name="action" value="Search">
> </P>
> </FORM>
>
> in the page http://tempuri.org/customer/customer.html. That will
> produce a form with one text input field and a button labelled
> "Search". If a user with an old, HTTP/1.0 browser entered "John Doe"
> in the Customer Name field and clicked the Search button, the browser
> would send a request similar to
>
> POST /customer/cgi/process HTTP/1.0
> User-Agent: your browser name here/some.version
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 26
>
> cust=John+Doe&action=Search
>
> That is, a POST HTTP request to the URL specified by the form's
> action attribute, with the form parameters encoded as name=value
> pairs, separated with "&" characters, URL-encoded (spaces become "+"
> signs and various special characters are hex-escaped), and sent as
> the request's content-body. Note that the content-body begins after
> a blank line following the headers, and the content-length does not
> include the blank line. RFC 1945 (HTTP/1.0) and the HTML 4.0 spec
> (see below) have the details.

This sounds good. If I just could fill in the text field
<input type="text" name="test" size="60" maxlength="20" class="text"
/>
<input type="submit" value="test!" class="submit" />
it would be something like
test=somerandomstring+anotherone&action=submit ?
the code just differs from yours. Here class-attributes are used.

I also found some footage from the site's html.
<!DOCTYPE html
        PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "DTD/xhtml1-transitional.dtd">
So, I guess it is standard 1.0 HTML.

> > C#, C, C++, Java, Perl, Python are all welcome... also any additional
> > libraries and/or APIs too.
>
> For something like this I'd typically just throw a hard-coded request
> with a couple of replacement fields (eg for Content-length) into a
> quick & dirty C program, but since you're not familiar with HTTP and
> HTML form encoding I'd definitely recommend looking for an API.
> Unfortunately I can't recommend any; I have my own already, so I
> haven't had to look for one. (The one I've written is part of a
> commercial software package, so I'm not free to share it.)

I'll start studying HTTP and HTML form encoding. Does usually HTTP
APIs contain these functionalities?

> If you do decide to go the route of building the request yourself,
> you can find all the relevant standards at http://w3c.org. You'll
> need the HTTP/1.0 spec (RFC 1945) for building the HTTP request and
> parsing the response, and one of the versions of the HTML spec (eg
> HTML 4.0) for the details on converting form controls to the proper
> request.