Re: Downloading lots and lots and lots of files

On 29 Jan 2007, coolneo@xxxxxxxxx wrote:

I managed to download about 21,000 of the 50,000 items over the course
of some time. Initally, Google was processing these items at a slow
rate but lately they have picked it up.

Bandwidth is indeed a concern, and I understand downloading 5TB will
take a long long time, but I think it would be a little shorter if I
could spawn off 4 downloads at a time, or even 2, during our off
business hours and the weekend (I get . The average file size is
125MB. We have a 200mb pipe, so it's not entirely unreasonable (is
it?).

You should contact Google and request the data directly. I guarantee
you they will be happy to avoid the load on their network and
servers, since HTTP is not the best way to transfer lots of data.

Ted
.

Relevant Pages

  • Re: HDTV Connection
    ... > | guarantee that every answer you want is right there, ... To give you an idea of how bad the Google search of AVS Forum is, ... All of this info (including source code downloads) is available ...
    (alt.tv.tech.hdtv)
  • Re: Problem with socketpair , AF_UNIX and select call - can anybody through any light on this!
    ... select only returns a hint that an IO operation ... might succeed, not a guarantee (google for "spurious wakeup"), hence a ...
    (Fedora)
  • Re: Monday
    ... Generally in advance and there's no guarantee that anyone will respect ... Usenet is an open medium. ... Google won't archive your post if you set the relevant flag AIUI. ...
    (uk.telecom.broadband)
  • Re: Google not archiving Borland newsgroups anymore?
    ... At 19:30:34, 25.04.2005, Rick Carter wrote: ... >>> Ever, no. Google shows them for a week or so, then removes them. ... The only guarantee that it won't show up is not posting at ... Prev by Date: ...
    (borland.public.delphi.non-technical)
  • Re: non sequential execution...
    ... s88 wrote: ... Most people do not use Google and there is absolutely no guarantee that they will have seen earlier messages in the thread. ... void sub(){ ...
    (comp.lang.c)