Trouble downloading pages


I've been writing and testing a program in Delphi 6 that is intended to
search through a huge fan-fiction archive (www.fanfiction.net), because
the search function there is no good. I've been using the Synapse TCP/IP
library to fetch pages before parsing the data.

So far it works fairly well, but there is one big problem that is really
aggravating me. Basically, some pages become inaccessible to my program
for long periods of time, for no apparent reason. While I am able to get
these pages fine in my web browser, my program gets a blank page or an
error page whenever it tries to download one of these pages.

The pages affected seem to be random (or almost random), and the period of
time that the condition lasts, ranges from a day or two to weeks, for a
particular page. It also persists when I use different versions
of my program, from different sites (including a port to Linux,
using Open Kylix 3). Recently, I have been getting this problem with
section pages, which is especially annoying.

I have no idea what could be causing this, so could anyone give me
suggestions on what could be cuasing this, and how I can get around it?

Thank You,
James Fox
.