Screaming Penguin

In a recent study by a South Dakota company, the Web is estimated to be 500 times larger than the estimate maps provided by the likes of Yahoo and Lycos. It seems there is now two distinct areas of the Web... the surface information that the search engines hit, and the "deep Web", which are areas much harder to find or access. It is now estimated that there are 550 billion documents stored on the Web. Compare this to the combined index of around 1 billion pages of the top search engines. Also compare this number to the Lycos index of 54,000 pages back in 1994. One of the biggest problems with search engines is their inability to index dynamically built pages built from databases. Search engine technology has not kept up with Web technology and is still only able to index static pages. There are a few companies that are currently working on applications that will do an extended search of the Web and deep Web, including database searches. Unfortunately, these searches can take anywhere from 15 to 90 minutes and can return an ocean of information.   CNET Article