Re: web crawling program

Or you could just make a textfile containing all the URLs and pass it
to WGET.
Or use Jerry's approach with CURL.

CURL would be my first approach since it's built-in, but you could use
WGET as well.
.

Relevant Pages

  • Re: Auto submitting a form that contains a file field
    ... Client end is NOT php. ...  Thanks, - Dave ... post-file in wget or the equivalent in CURL to do the thing. ...
    (comp.lang.php)
  • Re: Command-line file downloader
    ... including cURL and wget at ... The lack of recursive downloading is a pretty significant hit against ... OK, thats now rhetoric, but LFS support was introduced just recently. ...
    (microsoft.public.win2000.cmdprompt.admin)
  • Re: fetch extension - use local filename from content-dispositionheader (new diff)
    ... > use wget or curl instead? ... > get to endure such helpful attitudes? ... to accept patches to implement the feature you need for your scripts. ...
    (freebsd-current)
  • Re: [PHP] spider
    ... In another word: WGET ... But has anyone seen a PHP port of wget ... or is curl the only one of the 2 which does it natively in a compiled ... True, but some sites require natively running instead of running things via exec, system, etc... ...
    (php.general)
  • Re: [PHP] spider
    ... In another word: WGET ... Both are pretty effecitve and give pretty much the same results, ... But has anyone seen a PHP port of wget ... or is curl the only one of the 2 which does it natively in a compiled ...
    (php.general)