Week 10 Readings

David Hawking , Web Search Engines: Part 1 and Part 2 IEEE Computer, June 2006. http://www.computer.org/portal/site/computer/menuitem.5d61c1d591162e4b0ef1bd108bcd45f3/index.jsp?&pName=computer_level1_article&TheCat=1055&path=computer/homepage/0606&file=thingswork.xml&xsl=article.xsl&;jsessionid=G3nJ1tNBGT7hHN4hLhySlzJ4JnrH42kkmWrNhdrllpkGnMp2nC8N!-1182277384 and http://www.computer.org/portal/site/computer/menuitem.5d61c1d591162e4b0ef1bd108bcd45f3/index.jsp?&pName=computer_level1_article&TheCat=1055&path=computer/homepage/0806&file=howthings.xml&xsl=article.xsl&
I liked this reading. It is interesting how much more data search engines are indexing now than when they first started (1000 times as much). Many issues I had never given a great deal of thought to before, such as regarding crawlers: speed, "politeness," excluded content, duplicate content, continuous crawling, and spam rejection.

Among their 'Final Crawling Thoughts': Crawlers are highly complex parallel systems, communicating with millions of different Web servers, among which can be found every conceivable failure mode, all manner of deliberate and accidental crawler traps, and every variety of noncompliance with published standards... It is not uncommon to find that a crawler has locked up, ground to a halt, crashed, burned up an entire network traffic budget, or unintentionally inflicted a denial-of-service attack on a Web server whose operator is now very irate.

Part II. This section reviewed the algorithms and data structures needed for indexing huge amounts of web page text and delivering high-quality results millions of times a day. Although interesting, this part was more difficult for me to follow (definitely couldn't read it as quickly). I particularly liked their 'term lookup' capabilities. For speeding things up: skipping, early termination, clever assignment of document numbers, and caching.

Shreeves, S. L., Habing, T. O., Hagedorn, K., & Young, J. A. (2005). Current developments and future trends for the OAI protocol for metadata harvesting. Library Trends, 53(4), 576-589.

I think that the Open Archives Initiative is great. I agree with what they are doing and what their intentions are. Increasing the access to information is always a good thing.

MICHAEL K. BERGMAN, “The Deep Web: Surfacing Hidden Value” http://www.press.umich.edu/jep/07-01/bergman.html

Wow, this article is quite content-rich (not a quick read). I like his analogy to the ocean at the beginning: Searching on the Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it. A lot of interesting facts are brought up and discussed. (One example: Deep Web documents (13.7 KB mean size; 19.7 KB median size) are on average 27% smaller than surface Web documents.) Overall, this is a great article.