Re: building a meta search engine


"Dale King" <DaleWKing@xxxxxxxxx> wrote in message news:gtCdnaa0X54vmz_ZnZ2dnUVZ_v-dnZ2d@xxxxxxxxxxxxxxxx
RoS wrote:
Hello there,

I am building a web application, which involves submitting search
queries to a number of sites, processing and parsing search results and
returning them in an organized way.

Any thoughts/comments on the subject are greatly appreciated.

Have you verified that this is allowed by the sites you plan to use? If one of those sites is Google it definitely is not allowed.

You can get a (free, AFAIK) license from Google which will give you something like 1000 queries per day. They have code examples in Java showing how to access their search API.

To the OP: you should probably write an abstraction layer so that you can query each search engine using the same API. So with Google, you'd use their specific API and not worry about parsing HTML at all, and with other search engines, you'd do HTML or XML parsing, but it all looks the same to the calling class which just gets List<SearchResult> objects (or whatever), and deals with them.

- Oliver

.

Relevant Pages

  • Re: OT: Sandman and Snit come to an agreement. I welcome others to join.
    ... crashing down, they lost some big clients, and were hurting badly. ... Not all errors will cause Google problems, ... placed angle bracket kept Googlebot from correctly parsing ... bracket usually brakes parsing as well. ...
    (comp.sys.mac.advocacy)
  • Re: Need help changing a excel sheet to a access DB, NOT Linked
    ... When I thought more about it, I realized Access doesn't have "parsing" ... But I use queries to "parse"... ... Jeff Boyce ... <Access MVP> ...
    (microsoft.public.access.tablesdbdesign)
  • Re: Need to lock read a text file, then delete...
    ... Alessandro wrote: ... I tried to find some information about my subject with google, ... I guess we are talking about parsing a log file or something like that? ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Xml parser
    ... I want to know weather is there any api available in python for parsing ... xml(XML parser) ...
    (comp.lang.python)
  • memory management in swi prolog
    ... I test the performance of queries on UML models and compare it to a Java implementation. ... Therefore I iteratively execute the same queries on growing models (10,100,1000,... ... Therefore I have a question concerning that: is there a command to explicitly emtpy all stacks before I start parsing my XML file or some other possibility to clear up memory because it seems that for different queries the stack is filled differently and not emptied before parsing the next model. ...
    (comp.lang.prolog)