Re: Pocket Reference for Common Lisp

On Fri, 29 Feb 2008 07:32:25 -0800, Peter Christensen wrote:

Um, apparently more than just caffeine! I've been to that page probably
a dozen times, looked at, said "There's no way I'm downloading all these
files, I wish there was a package" and then left. Anyone, feel free to
chime in and tell me that "There's no such things as a stupid question"!

Thanks,
Peter

Not needed in this case, but there is always wget, on *nix at least. I
recently used it to download over 20,000 files from a web site, over a
period of several weeks. It even has restart facilities.

Tim

WGET(1)
GNU Wget
WGET(1)

NAME
Wget - The non-interactive network downloader.

SYNOPSIS
wget [option]... [URL]...

DESCRIPTION
GNU Wget is a free utility for non-interactive download of files
from the Web. It supports HTTP, HTTPS, and FTP protocols, as well
as retrieval through HTTP proxies.

Wget is non-interactive, meaning that it can work in the
background, while the user is not logged on. This allows you to
start a retrieval and disconnect from the system, letting Wget
finish the work. By contrast, most of the Web browsers require
constant user's presence, which can be a great hindrance when
transferring a lot of data.

Wget can follow links in HTML and XHTML pages and create local
versions of remote web sites, fully recreating the directory
structure of the orig- inal site. This is sometimes referred to as
``recursive downloading.'' While doing that, Wget respects the
Robot Exclusion Standard (/robots.txt). Wget can be instructed to
convert the links in downloaded HTML files to the local files for
offline viewing.

Wget has been designed for robustness over slow or unstable network
connections; if a download fails due to a network problem, it will
keep retry- ing until the whole file has been retrieved. If the
server supports regetting, it will instruct the server to continue
the download from where it left off.
.

Relevant Pages

  • Re: problem with wget
    ... I want to download a document based on my web server, ... wget is able to detach itself from shell and put in the background. ... Is there any wait to download a http document in C language by an other way? ...
    (comp.unix.programmer)
  • Re: large-scale vector data ftp & Re: looking for places data
    ... HTTP at http://eros.usgs.gov/geodata/dlg_large/states.html ... I don't know where they might be available through FTP. ... for example you can run wget to download a URL that you specify on ...
    (comp.infosystems.gis)
  • wget seems to not be able to download *.shtml is that correct? (probably not, please help)
    ... I would like to download several files from a particular web site. ... In the past I have just pointed wget at it and there were no problems. ... I can download any of the files with a web browser, ...
    (comp.os.linux.misc)
  • Re: Free Metalworking Plans
    ... NRA LOH & Endowment Member, Golden Eagle, Patriot"s Medal. ... | I Dl'd the lathe plans, ... Use something like Wget ... or some other worthy download manager to retrieve it. ...
    (rec.crafts.metalworking)
  • Wget usage : request for comments
    ... I am going to start a small project to analyze ... 8 websites with hyperlinks, images, js, etc.. ... I will use wget as a crawler (I like command ... seems I can't download the results in text format. ...
    (comp.os.linux.misc)