Volunteers based search project to take on Google?
- Thread starter temi
- Start date
D
Duke
Guest
Just out of curiosity, how exactly is this going to work anyway? If I take dmoz as an example, they have plenty of links to dead sites as do many directories I've run across. With the ever changing dynamic of the www., how can a human edited Search Engine hope to stay current with every site in its listing?
Duke, it isn't human edited, it is human seeded, there is a difference. The initial seeding will be done manually. This is wjhat Zoltan Gyongyi said in the original study named
Combating Web Spam with TrustRank. Technical Report, Stanford University, 2004
Web spam pages use various techniques to achieve higher-than-deserved rankings in a search engine's results. While human experts can identify spam, it is too expensive to manually evaluate a large number of pages. Instead, we propose techniques to semi-automatically separate reputable, good pages from spam. We first select a small set of seed pages to be evaluated by an expert. Once we manually identify the reputable seed pages, we use the link structure of the web to discover other pages that are likely to be good. In this paper we discuss possible ways to implement the seed selection and the discovery of good pages. We present results of experiments run on the World Wide Web indexed by AltaVista and evaluate the performance of our techniques. Our results show that we can effectively filter out spam from a significant fraction of the web, based on a good seed set of less than 200 sites.
Original doc here http://dbpubs.stanford.edu:8090/pub/2004-17
Combating Web Spam with TrustRank. Technical Report, Stanford University, 2004
Web spam pages use various techniques to achieve higher-than-deserved rankings in a search engine's results. While human experts can identify spam, it is too expensive to manually evaluate a large number of pages. Instead, we propose techniques to semi-automatically separate reputable, good pages from spam. We first select a small set of seed pages to be evaluated by an expert. Once we manually identify the reputable seed pages, we use the link structure of the web to discover other pages that are likely to be good. In this paper we discuss possible ways to implement the seed selection and the discovery of good pages. We present results of experiments run on the World Wide Web indexed by AltaVista and evaluate the performance of our techniques. Our results show that we can effectively filter out spam from a significant fraction of the web, based on a good seed set of less than 200 sites.
Original doc here http://dbpubs.stanford.edu:8090/pub/2004-17
D
Duke
Guest
I'm missing the point somehow OWG so I sincerely apologize for my ignorance. I think I'll take some time this weekend and give the link a good long read and maybe post back here with further questions.
I guess what doesn't add up for me is by the sounds of things, at some point a human will have to be involved in quantifying either a page, search result or some other info., and with the billions of sites online and more added every minute of each day, I just don't see how this can work.
Anyway, I was never a math genious so if I'm rambling just tell me to stfu.
I guess what doesn't add up for me is by the sounds of things, at some point a human will have to be involved in quantifying either a page, search result or some other info., and with the billions of sites online and more added every minute of each day, I just don't see how this can work.
Anyway, I was never a math genious so if I'm rambling just tell me to stfu.
Ok Duke,
Just for you I have knocked this flow chart up to better explain it.
Please not that this image is copyrighted
Just for you I have knocked this flow chart up to better explain it.
Please not that this image is copyrighted
This illustration helps explain trust ranking better, questions, what sites apart from the one below will fall within the category of trusted site? Are trusted sites the same site link vendors call authority site?
I can think of the following sites as Trusted site, which other sites fits the bill or which of the one I listed should be removed?
dmoz
yahoo directory
institution like BBC
Government website
Major newspaper sites
Banks
what other sites?
I can think of the following sites as Trusted site, which other sites fits the bill or which of the one I listed should be removed?
dmoz
yahoo directory
institution like BBC
Government website
Major newspaper sites
Banks
what other sites?
Temi, My understanding is that they will not just say 'major newspapers' after all in the UK the main newspapers are the sun and the news of the world, both of which will link to any old crap. Certainly they might look at the newspapers, and decide that the Timnes, FT, Guardian, Observer are ok, but the daily sport, daily mirror sun etc are junk.
I would also say that google might split Dmoz into areas that are trusted, and areas that are not. It would not surprise me to see DMOZ devalued to almost nothing. It would also not surprise me to see google make an offer to Netscape for DMOZ to use as their main seed.
I would also say that google might split Dmoz into areas that are trusted, and areas that are not. It would not surprise me to see DMOZ devalued to almost nothing. It would also not surprise me to see google make an offer to Netscape for DMOZ to use as their main seed.
Thanks very much for that OWG,
Yes, the though of Sun being regarded as a trusted authority site is frightening When I think of newspapers, Sun, Mirrors and other tabloid don't cross my mind at all.
I some how cannot see Google making and offer for DMOZ , they can already get the contents free of charge, starting their own directory is a much feasible/better proposition that buying DMOZ
Yes, the though of Sun being regarded as a trusted authority site is frightening
When I think of newspapers, Sun, Mirrors and other tabloid don't cross my mind at all.I some how cannot see Google making and offer for DMOZ , they can already get the contents free of charge, starting their own directory is a much feasible/better proposition that buying DMOZ
If they buy DMOZ, then remove all the editors, employ their own, they can control the content of it.
Yes, but why bother doing that? it will be unpopular, give them some bad name saying they are throwing their weight about etc etc when is VERY easy to start their own directory from scratch an use a better system from the ground up. Like they created Google talk instead of buying the over priced Skype
R
Robster
Guest
So getting back on topic.......
The about page doesn't give much detail about how it works, but looking further into the site it seems the crawling is done through distributed computing, but the index is held on a central server/servers.
Their growth looks impressive
http://www.majestic12.co.uk/projects/dsearch/stats.php
The about page doesn't give much detail about how it works, but looking further into the site it seems the crawling is done through distributed computing, but the index is held on a central server/servers.
Their growth looks impressive
http://www.majestic12.co.uk/projects/dsearch/stats.php
Similar threads
