Re: urllib - changing the user agent

From: Samuel Walters (swalters_usenet_at_yahoo.com)
Date: 01/09/04 

Date: Fri, 09 Jan 2004 21:17:24 GMT

|Thus Spake John J. Lee On the now historical date of Fri, 09 Jan 2004
20:16:54 +0000|
 
> or again, you can set .addheaders on OpenerDirector (which will cause
> those headers to be added to all requests).

This, however, does not stop the original User-agent header to be sent,
and google still filters out the request. Instead, it just causes a
second user-agent to be sent.

Here is the very bad code I used to solve this problem. There are better
ways, I assure you, but it should point you in the right direction. You
should probably do this with inheritance, but for my quick script, this is
what I did. 

----
#Override the default OpenerDirector Class Init.
#OpenerDirector *insists* on adding a User-Agent 
#That some websites don't like.
foo = OpenerDirector.__init__
def bar(self, Agent='None'):
         foo(self)
         self.addheaders = [('User-Agent',Agent)]
 
OpenerDirector.__init__ = bar
-----
HTH
Sam Walters.
-- 
Never forget the halloween documents.
http://www.opensource.org/halloween/
""" Where will Microsoft try to drag you today?
    Do you really want to go there?"""

Relevant Pages

  • Re: OT - Blue Frog
    ... Blue Frog doesn't use the headers to determine where to send opt out ... If these opt-out requests crash their servers it's because of one of ... They must scale their solicitations down to be commensurate ...
    (uk.media.radio.archers)
  • Re: Why not serve XML documents with xml-stylesheet PIs? (Re: XML Web Site + MySQL)
    ... If they lie in such a way that the correct ... User-Agent header doesn't contain all that much useful ... Yes, in controlled environments placing ...
    (comp.text.xml)
  • Cache-control and Expires headers
    ... I'm using a wildcard mapping so that all requests go via ... Then I'm using an HttpModule to look at all the requests and do some ... either no Expires header, or at least something with a long expiration date, ... PostRequestHandlerExecute) nothing happens since the headers have already ...
    (microsoft.public.dotnet.framework.aspnet)
  • Parse tcpdump for HTTP Request Response Headers
    ... writing out the headers to different files based on the server program ... response headers for successful requests with query strings. ...
    (comp.lang.perl.modules)
  • Re: do you get error in this url?
    ... someone had pointed out something about user-agent.. ... So next line, sBrowser actually is a null pointer, and dereferencing it throws a NPE. ... You told us one of your customers triggered the error, but you've tried to reproduce it with a lot of common browsers, in a lot of situations, and couldn't reproduce. ... client HTTP headers. ...
    (comp.lang.java.help)