Fetching websites with Python

From: Markus Franz (mf_at_orase.com)
Date: 03/31/04 

Date: Wed, 31 Mar 2004 19:33:45 +0200

Hi.

How can I grab websites with a command-line python script? I want to start
the script like this:

./script.py ---xxx--- http://www.address1.com http://www.address2.com
http://www.address3.com

The script should load these 3 websites (or more if specified) in parallel
(may be processes? threads?) and show their contents seperated by ---xxx---.
The whole output should be print on the command-line. Each website should
only have 15 seconds to return the contents (maximum) in order to avoid a
never-ending script.

How can I do this?

Thanks.

Yours sincerely

Markus Franz

Relevant Pages

  • Re: Integrated Auth / Default Domain Issues
    ... Then you can easily run the script against multiple machines (or even multiple websites on the one machine). ... Do you think it would also be possible to use this same tool to keep a second copy of the 'same site' in sync, but without the NTLM authentication enabled? ...
    (microsoft.public.inetserver.iis.security)
  • [Full-Disclosure] scriptkids on this list
    ... It seems to me that since you point out to efnet.ru websites, ... shut up you script kiddies... ... > Stevenson, Ron DuFresne, and Nuno Fernandes calling people -- people ... > to defend their pride -- pride which is obviously at gunpoint when ...
    (Full-Disclosure)
  • Re: VBScript halts with out error message
    ... If I exit the script and run the batch file it ... The command lines are based on IIS websites. ... Const ForWritting = 2 ...
    (microsoft.public.scripting.vbscript)
  • RE: [Full-disclosure] SecNiche : Microsoft Internet Explorer Pop up Blocker Bypassing and Dos Vu
    ... I don't see anything in the script that can bypass zone security and run ... drawn conclusion that the script can execute from internet zone. ... Microsoft Internet Explorer Pop up Blocker Bypassing and Dos ... registry entries for specific websites through Javascript. ...
    (Bugtraq)
  • Re: ipfw - log one computers usage
    ... I'd like it also to resolve the names of websites that were ... Squid or some other proxy server would make this easier but a shell ... script that did some cats, greps, and awks could do it as well. ...
    (comp.unix.bsd.freebsd.misc)