url question - extracting (2 types of) domains
- From: lazy <arunmail@xxxxxxxxx>
- Date: 15 May 2007 19:04:02 -0700
Hi,
Im trying to extract the domain name from an url. lets say I call
it full_domain and significant_domain(which is the homepage domain)
Eg: url=http://en.wikipedia.org/wiki/IPod ,
full_domain=en.wikipedia.org ,significant_domain=wikipedia.org
Using urlsplit (of urlparse module), I will be able to get the
full_domain, but Im wondering how to get significant_domain. I will
not be able to use like counting the number of dots. etc
Some domains maybe like foo.bar.co.in (where significant_domain=
bar.co.in)
I have around 40M url list. Its ok, if I fallout in few(< 1%) cases.
Although I agree that measuring this error rate itself is not clear,
maybe just based on ituition.
Anybody have clues about existing url parsers in python to do this.
Searching online couldnt help me much other than
the urlparse/urllib module.
Worst case is to try to build a table of domain
categories(like .com, .co.il etc and look for it in the suffix rather
than counting dots and just extract the part till the preceding dot),
but Im afraid if I do this, I might miss some domain category.
.
- Follow-Ups:
- Re: url question - extracting (2 types of) domains
- From: Michael Bentley
- Re: url question - extracting (2 types of) domains
- From: Gabriel Genellina
- Re: url question - extracting (2 types of) domains
- Prev by Date: Re: PEP 3131: Supporting Non-ASCII Identifiers
- Next by Date: Re: Interesting list Validity (True/False)
- Previous by thread: Re: Spotting Crashed Application
- Next by thread: Re: url question - extracting (2 types of) domains
- Index(es):
Relevant Pages
|
