Crawler algorithms have a 'politeness delay' so that websites/pages aren't bombarded .
Friday, November 7, 2008
Week 11 Notes
The Deep Web has about 500 times the amount of info as the surface web. This is so interesting, as most people are unaware of this, operating strictly on the surface level. Most search engines 'crawl' over the surface of the web picking out and ranking the most popular pages. Pages under the surface are virtually invisible to the web crawlers, as they do not have the requisite links that would ensure their popularity.
Subscribe to:
Post Comments (Atom)
2 comments:
I particularly liked the pictures of the fishing boats. The net skimming the surface and the lines going deep. The nets capture so much more than the lines and the depths of the ocean are imense. Kind of puts the deep web into perspective. Deep, dark and a bit scary
Hi Alberta!
My thing about Deep Web is if you don't know about it, how do you get to it? You can't search it. You must have to use six degrees of separation, but it still requires that the Web Site has a friend. How frustrating to know stuff is out there, but not be able to get to it. Unless you're in the know.
By the way I think Chuck Palahniuk is pretty darn cool too. I read Rant and was hooked. Hope all is well.
Post a Comment