Re: How to write something to a html textfield and send it?

From: Michael Wojcik (mwojcik_at_newsguy.com)
Date: 03/11/04 

Date: 11 Mar 2004 17:02:26 GMT

In article <f53a6ce0.0403100530.5476c211@posting.google.com>, alexhanh@ranssi.paivola.net (Alexander) writes:
>
> I have a standard html web-page with a textfield and a send button
> added to it (look below for html). I need a way to open the page, find
> the specific textfield, write something to it and "click" send. I need
> to do this through a program.

Do you actually need to control a particular browser and make it
perform these actions, or do you need to write an HTTP user agent
which can submit the form with the text field populated?

And do you need a general solution for this problem, or is it only
for this one form? And if the latter, how confident are you that
the form won't change significantly during the time you need this
application to run?

I've never had any desire to automate a browser, but it can certainly
be done for most of the browser / UI combinations I'm familiar with.
However, it's usually much easier to write a small HTTP/1.0 user
agent to post the request and retrieve the response. (An HTTP/1.1
user agent is more work, since even a conditionally-compliant
HTTP/1.1 UA must support the chunked transfer-encoding.)

> I know that perl has some nice socket/http functions, which can be
> used easily to detect the html tags and the required textfield. But
> what about sending some data to the textfield and clicking the button?

Again, the question is whether you need the browser at all. Do you
need to write to the text field control in the browser and click the
button, or do you just need to submit the request with the appropriate
text?

> And no, I'm not able to do it through calling a prefixed www-address,
> like http://www.google.com/search?q=problem to search for "problem".

Most non-trivial HTML forms use HTTP POST requests. The URL in your
example contains a query-string, which is produced by a user agent
for a GET request.

They're rather different mechanisms; with POST, the request parameters
are passed as the HTTP content-body of the request message, in URL-
encoded form. You take the parameter names and their values, URL-
encode them, and put them after the HTTP header and separator in the
request.

Here's a quick example. Say there's a page with a form like

   <FORM action="cgi/process" method="POST">
      <P>
         Customer Name: <INPUT type="text" name="cust">
         <INPUT type="submit" name="action" value="Search">
      </P>
   </FORM>

in the page http://tempuri.org/customer/customer.html. That will
produce a form with one text input field and a button labelled
"Search". If a user with an old, HTTP/1.0 browser entered "John Doe"
in the Customer Name field and clicked the Search button, the browser
would send a request similar to

   POST /customer/cgi/process HTTP/1.0
   User-Agent: your browser name here/some.version
   Content-Type: application/x-www-form-urlencoded
   Content-Length: 26

   cust=John+Doe&action=Search
   
That is, a POST HTTP request to the URL specified by the form's
action attribute, with the form parameters encoded as name=value
pairs, separated with "&" characters, URL-encoded (spaces become "+"
signs and various special characters are hex-escaped), and sent as
the request's content-body. Note that the content-body begins after
a blank line following the headers, and the content-length does not
include the blank line. RFC 1945 (HTTP/1.0) and the HTML 4.0 spec
(see below) have the details.

> C#, C, C++, Java, Perl, Python are all welcome... also any additional
> libraries and/or APIs too.

For something like this I'd typically just throw a hard-coded request
with a couple of replacement fields (eg for Content-length) into a
quick & dirty C program, but since you're not familiar with HTTP and
HTML form encoding I'd definitely recommend looking for an API.
Unfortunately I can't recommend any; I have my own already, so I
haven't had to look for one. (The one I've written is part of a
commercial software package, so I'm not free to share it.)

If you do decide to go the route of building the request yourself,
you can find all the relevant standards at http://w3c.org. You'll
need the HTTP/1.0 spec (RFC 1945) for building the HTTP request and
parsing the response, and one of the versions of the HTML spec (eg
HTML 4.0) for the details on converting form controls to the proper
request. 

-- 
Michael Wojcik                  michael.wojcik@microfocus.com
But I still wouldn't count out the monkey - modern novelists being as
unpredictable as they are at times.  -- Marilyn J. Miller