Search Engines

On this page:
What Are Search Engines? | Other Types of Internet Search Tools | Problems With Search Engines | More Search Engines | Invisible Web

What Are Search Engines?

A search engine is a tool that explores its own database of text for Web pages that contain specific words, phrases and/or numbers that match the specific terms you requested. When found, "click-able" links to these pages are presented in a list ranked by relevancy.

Search engines work by sending programs, called "spiders" or "robots" that constantly "crawl" the Web, looking at Web pages, following the links they find and add any new pages found to its master database or index. It is this master database that is searched when you make a request.

Some search engines look at the entire document for important words, others only look at the first few sentences or "metatags" (descriptive words used by the Web page creator to summarize the content of the page) or just the title.

Once found, all the links are ranked or arranged according to rules that vary from search engine to search engine. The important factors tend to be how often or how soon the sought for term appears, whether the term appears in the title or unseen metatag, or how unique the word(s) is. Some Web page creators try to manipulate these ranking procedures in order to get their pages to appear higher on the results list. Some engines attempt to guess what the user was looking for by guessing at the context of the terms. Other search engines rank sites by the number of other sites that link to it and how popular those sites are.

Other Types of Internet Search Tools

Subject Directories

Subject directories are created and maintained by real, live people, not electronic "spiders". Humans find, review, annotate and organize Web sites according to a predetermined hierarchy of subject categories, sub-categories, etc. There is usually a search engine that can search this hierarchy. To cut costs, some directories no longer look for appropriate sites, but charge to include submitted sites.

Because of this human intervention, listed Web sites tend to be of a higher quality, with fewer out of context results. But, since a directory doesn't actually store the pages, only the pointers to other sites, those Web pages may move, change or disappear. Directories are excellent for browsing, especially of general topics and commercial sites.

Examples of non-commercial directories are Librarian's Internet Index - "Information You Can Trust." (http://lii.org/ ) compiled by librarians at the University of California Berkeley and the VRL - Virtual Reference Library, sites selected by staff of Toronto Public Library.

Portals

Portals are commercial sites that, in addition to the list of subject categories, offer current news, weather, stock quotes, e-mail and other services in order to attract repeat users and thus sell more advertising, e.g. Excite, Yahoo!

Meta Search Engines

Meta search engines do not have their own searchable databases. Instead, they send your search enquiry simultaneously to a set of individual search engines, from a single site and using the same interface.

Most meta search engines present the results in a single merged list, from which duplicate entries have been removed. A few meta search engines display results in separate lists as they are received from each engine. Duplicate entries are not removed.

Meta searcher engines provide a fast way of finding out which engines are retrieving the best results for your search. Use them to obtain a quick overview on a subject and/or unique term, especially when you are not having any luck pulling up documents in your search.

When you initiate a keyword or phrase search on a meta search engine, you seldom have a choice on how the search is configured and conducted. Meta search engines don't offer the search options that individual search engines do.

Although meta search engines claim to query about 10 - 20 individual search engines, many results are duplicated, subject directories or pay-per-click engines. Few query Google, the largest and most popular search engine on the Web.

Examples of Meta Search Engines

• eZ2Find - Searches matching Web sites, results from the Invisible Web and links related clusters for subject targeted searching.
• Fazzle - Target searching to Canadian content, downloads, images or video. Also supports Boolean queries and allows you to limit your search to a phrase, title, or URL.
• Ixquick - Ranks results based on the number of "top 10" rankings a site receives from its various search sources.
• KartOO - Search results are presented on a map.
• MetaCrawler (http://www.metacrawler.com) - Searches Web directories, sponsored listings, all the major search engines and specialized databases.
• Clusty - A major trend among meta searchers is the development of document clustering. Vivisimo is one of the best. Instead of ranking results, it automatically organizes the links into categories.

Trends

Recent trends is the visualization of the data. For some people seeing how one website relates to other sites is useful. A fun site is www.webbrain.com. It is an active directory. Clicking on one of several general categories, causes them to move around, displaying new, more specific categories. This allows you to drill down many levels. The bottom third of the screen displays links appropriate to the category open above.

Problems With Search Engines

1. Volume - there are about 3 billion Web pages available to the public, and this number is increasing every year. It takes days or even weeks to crawl and record the entire Web. For this reason, search engines aren't useful for finding recent information or new Web sites.
2. Words can have more than one meaning, e.g. "Mercury" is a car, a planet, a Roman god and a chemical element.
   For these reasons, a search request that isn't specific enough will return millions of "hits" or possible links to Web pages.
3. Search engine "robots" or "spiders" can only follow links - they are unable to gain entry to databases of facts that require typing. Nor can they "see" graphics. Thus, about 2/3 of the WWW is not indexed.
4. Structuring an effective search strategy - some people find it difficult or frustrating to express their questions in a structured way in order to produce a successful search. Also, try to think of different words or phrases to describe your concept.
5. Spelling counts - however, many search engines will suggest alternate spellings!
6. Trustworthiness - unlike books or magazines, Web pages are not reviewed or edited. Thus, the quality of the information varies - user beware.
7. Search engines that rank links to Web sites based on their popularity - may distort their importance. The more people that look at the site, the higher it may be ranked, causing more people to see it and causing it to be ranked higher!
8. Pay for Placement - some webmasters are willing and able to buy your attention. Some search engines and directories clearly list "paid for listings" as "sponsored", other descriptions are more ambiguous. When portals and metasearch engines combine the results of multiple search engines, the distinction between relevant and paid for sites disappears. Did the site come up because it was better than the other sites, or because they paid more?

Consult the search engine's HELP in order to understand and make use of word truncation, Boolean operators, phrase searching and nesting for more effective searching.

For these reasons a Web directory of sites chosen and categorized by subject specialists is often more successful at finding information, e.g. BPL Recommended Sites.

However, a search engine can be useful for finding distinctively named organizations or very specific topics that won't appear in a lot of Web sites.

Even More Search Engines

• Search Engines - Google provides a listing of different search engines.
• Family Friendly Search Engines - There are a variety of "Family Friendly" search engines, again, provided by Google.

Invisible Web

The "Visible Web" is what you see in the results pages from general Web search engines. It's also what you see in almost all subject directories. The "Invisible Web" is what you generally cannot retrieve (see) in search results and other links contained in these types of tools.

How do you find this information?

Use searchable databases. Most of the invisible Web is made up of the contents of thousands of specialized searchable databases that you can search via the Web. Search results from many of these databases are delivered to you in Web pages that just display your search. Such pages very often are not stored anywhere; it is easier and cheaper to dynamically generate the answer page for each query than to store all the possible pages containing all the possible answers to all the possible queries people could make to the database. Search engines cannot find or create these pages.

Examples

• Burlington Public Library offers a number of databases such as EBSCOhost that you access abd search through our Web site. Some databases will allow you to search from your home computer, however you will require your library card number. See BPL's Electronic Databases.
• LII.org will direct you to a number of searchable databases.
• Recipe sites, such as Epicurious.com generate recipes from its database.

Updated March 2010

Home | Library Internet Policies | Change Text Size | Site Index

Copyright © 2010, Burlington Public Library, Ontario, Canada - Copyright & Disclaimer