Re: building a meta search engine


"Dale King" <DaleWKing@xxxxxxxxx> wrote in message news:gtCdnaa0X54vmz_ZnZ2dnUVZ_v-dnZ2d@xxxxxxxxxxxxxxxx
RoS wrote:
Hello there,

I am building a web application, which involves submitting search
queries to a number of sites, processing and parsing search results and
returning them in an organized way.

Any thoughts/comments on the subject are greatly appreciated.

Have you verified that this is allowed by the sites you plan to use? If one of those sites is Google it definitely is not allowed.

You can get a (free, AFAIK) license from Google which will give you something like 1000 queries per day. They have code examples in Java showing how to access their search API.

To the OP: you should probably write an abstraction layer so that you can query each search engine using the same API. So with Google, you'd use their specific API and not worry about parsing HTML at all, and with other search engines, you'd do HTML or XML parsing, but it all looks the same to the calling class which just gets List<SearchResult> objects (or whatever), and deals with them.

- Oliver

.

Relevant Pages