Re: web scraping

Alex Goor wrote:
I was hoping to write a simple program (if that's possible) to open a
browser, go to a site, and scrape a piece of information from that
site.

For example, I was hoping to open a Safari of Firefox browser, go to
nyt.com and scrape the Dow Jones Industrial Average which is on the
homepage.

Does anyone know where I could get an example program that does this
kind of thing to teach myself the concepts?

Driving an actual Web browser is awkward and unnecessary unless the page you want cannot be handled with a Perl module.

Take a look at WWW::Mechanize and see if it suits your purpose.

Rob
.

Relevant Pages

  • Re: Python Screen Scraper
    ... I am using Firefox. ... so I would like to scrape the result page for each ... interface python with Firefox, forwarding scraped pages to my browser ...
    (comp.lang.python)
  • Python Screen Scraper
    ... I have a tendency to go past my peak for training efficiency, so I would like to scrape the result page for each problem I answer, compile statistics, and have a program alert me when I should stop. ... Parsing, etc, is not a problem, but I'm not exactly sure how I might interface python with Firefox, forwarding scraped pages to my browser. ...
    (comp.lang.python)
  • Re: Python Screen Scraper
    ... I am using Firefox. ... so I would like to scrape the result page for each problem I ... interface python with Firefox, forwarding scraped pages to my browser ...
    (comp.lang.python)
  • web scraping
    ... For example, I was hoping to open a Safari of Firefox browser, go to nyt.com and scrape the Dow Jones Industrial Average which is on the homepage. ...
    (perl.beginners)
  • scraping email addresses
    ... Do spammers scrape the email address from both the output page in your ... browser and the HTML source? ... lynx -source -dump ...
    (comp.infosystems.www.servers.mac)