Searching the Deep Web

I recently developed a private workshop on how to find deep web resources on a specialized topic, and realized that the secret to finding information in the deep web is know that it’s a very different experience than searching the open web. While deep web content isn’t indexed by search engines, you can use search engines to find pointers, leads and links to deep web resources. Even more than with most “traditional” searches, looking for deep web content means thinking like a detective — looking for clues, using your peripheral vision to notice references or footnotes, and knowing when to step back and reassess. Following are some of the key approaches I recommend for finding deep web content on a particular topic.

Use a search engine to find a page that is describing a relevant database by adding terms likely to appear the page. To find content on income inequality, for example, you might use a query like “income inequality” (database OR “data set” OR archive OR databank).

Look for a libguide or other finding tool developed by a librarian or info pro. Since most of these resources include the word guide or libguide in the URL, you can find a useful libguide on, say, entomology, by searching entomology (inurl:libguides OR inurl:guides OR inurl:researchguides).

Follow the bread crumbs. Start with one known resource that you can identify — an association, government agency or non-profit — and then all the links from that site to other resources. In addition, search for mentions of that resource. For example, if you find that FAOSTAT, the statistics portal of the UN Food and Agriculture Organization, is a useful source, try searching for mentions of its URL with a query like “fao.org/faostat/” (including the quotation marks).

Leave a Reply