Intology – Intelligent Technology News
Computers Technology Internet Arts Business Science Sports

The multitude of search engines that are present today make searching the Web for business information a very easy task and seem to present access to a large amount of information. However, a quantum of information many times greater exists which is inaccessible using conventional search engines.

This portion of the web has been termed the Invisible Web, a term coined in 1994 by Dr. Jill Ellsworth. Frank Garcia in 1996 elaborating on the Invisible Web wrote that “It would be a site that’s possibly reasonably designed, but they didn’t bother to register it with any of the search engines. So, no one can find them! You’re hidden. I call that the invisible Web.”

A study conducted by BrightPlanet, a South Dakota company that has developed new software to plumb the Internet’s depths, finds that the Web contains over 550 billion documents with search engines indexing only about 1 billion pages. The study according to many researchers implies that over 80% of the web is today part of the Invisible Web.

Conventional search engines use highly advanced automatic programs called spiders to search the web and make it accessible. The indexes that these spiders create however do not include pages which change dynamically, have password protection or are specially designed to avoid detection. Documents containing text, graphics, CGI scripts, Macromedia flash or PDF files also cannot be indexed unless a link is provided to them. Conventional search engines like Google and Yahoo have started designing spiders to address these issues. They still have a long way to go before content that is presently part of the Invisible Web could be made accessible.

The majority of data in the Invisible Web is made up by Medical databases, Discussion lists, Patent databases, Phone numbers and personal contact databases, Government databases, Scientific databases, Auction databases, Legal databases, Dictionaries and thesauri, Knowledge databases and Product catalogs. These present a very large amount of information that could make things easier for many users of the internet.

The Invisible Web could give immediate access to a piece of information that a researcher has spent hours trying to find using conventional search engines. This need has been realized by many companies and organizations which have come up with special tools and search engines intended to allow access to the Invisible Web.

A few of these special search engines are:

1.    Complete Planet at www.completeplanet.com provides an extended listing that includes those of conventional search engines as well as the Invisible Web’s searchable databases. It is most effectively used to search not just browse listed resources.

2.    Deep Dyve at www.deepdyve.com gives easy quick access to resources within the invisible web like databases.

3.    A couple of the others include InvisibleWeb.com, and the Lycos Directory of Searchable Databases at http://dir.lycos.com/Reference/Searchable_Databases/

It would be an impossible task to list all the resources that are being created to make the Invisible Web accessible to people. As a matter of fact, it is these resources that are also making it more difficult to draw the line between the visible and the Invisible Web. This is because once a resource is made accessible it no longer remains in the realm of the Invisible Web.