Re: Trouble downloading pages

On Mon, 30 May 2005 08:24:26 +1200, Nicholas Sherlock wrote:

> James Fox wrote:
>> I've been writing and testing a program in Delphi 6 that is intended to
>> search through a huge fan-fiction archive (www.fanfiction.net), because
>> the search function there is no good. I've been using the Synapse TCP/IP
>> library to fetch pages before parsing the data.
>
> Some servers block spiders which are doing the same thing you are doing
> (Basically indexing the whole site), since they chew through expensive
> bandwidth. This may be the problem.
>
> Cheers,
> Nicholas Sherlock

Well, if so, the way FanFiction.Net is going about it is rather weird,
since only certain pages are blocked (out of a huge set), for a certain
period. However, the pages are blocked even if I enter their urls into a
separate Delphi test program using the Synapse Library, so I am wondering
if there might be something about the library which is off, or if I could
somehow tweak the library to disguise my queries (I've already tried
changing the user-agent string, but that doesn't seem to work). Does
anyone have any suggestions?

James Fox

.