Demystifying Search Engines
Search engines are complex and dynamic, constantly changing how they
search and what they search. To keep up with all the changes is a job
in itself. The article below will hopefully help you understand them a
How important is being listed (indexed) with search engines? See the
following statistics below.
- 85% Web surfers use search engines to find sites
- 91% Web surfers will change their queries if they are dissatisfied
with the first 30 hits
- The first site listed after a search receives three times more hits
than the fifth site listed
- The first 10 hits are visited 78 per cent more often than sites
listed 11th to 30th
- Only 30% of corporate Web sites have basic meta-tagging (see http://www.suestudios.com/gloss.htm#M
on our website for definition)
- Only 7 % of corporate Web sites are optimized for indexation
Search Engine Components
A Search Engine has 3 basic parts:
Spider (crawler, robot, worm or link finder): a computer program
that roams the Web looking at HTML documents, collects data about them
and brings back, sorts and then indexes the results. The data the spider
collects will be a summary of keywords and a description of the page.
Index: a database of the results collected by the spider that
is organized and searchable. When using a search engine, you are actually
searching the database, not "the Internet". The index is NOT organized
by subject categories but are ranked by a computer algorithm.
Search and retrieval mechanism: Software that allows users
to search the Index and return results in a predetermined order. This
component is the interface (the program you actually use when using
a search engine) between the end-user and the database.
Search engines vary as to the:
Number of sites they index. Not even the biggest search
engines come close to searching the entire web.
Frequency of updating or "refreshing" the database.
Methods of searching. The same search with different engines
will yield different results.
Versatility and complexity of the interface.
Order of the results list. Some engines look at the number
of pages that link to the site and incorporate this "popularity"
component into the mix.
Parts of each website they index (varies from URL and title
to full-text of every web page). Some index words in the title,
URL, introductory paragraphs, or full-text of all documents on a
web site. Some use a combination of these words and phrases, all
of which are entered into the search engine's database.
Getting Listed on Search Engines
If you want to make your Web site as profitable as possible, Web surfers
have to be able to find it using search engines and directories. If you
can manage to get a good ranking for some of your top key phrases, you
can get a steady stream of traffic.
Manually submitting to search engines is usually done by visiting
their front page and scrolling down to the bottom of the page where
you will see a link titled 'Add URL' , 'Add Your Site'
or 'Submit a Site'. It must be done directly at each search
engine. This can be a very time consuming and complex exercise,
as each search engine has different submission (indexation) guidelines.
For example, search engines will vary on the number of keywords
you can submit, how they are formatted and the length of the description
of the site. Free submissions to search engines will vary the length
of time that your site will actually be listed. Some will take months
to actually be listed.
Concentrate on being listed in the top 15 search engines: Yahoo,
MSN, AOL, Lycos, Google, Netscape, Excite, iWon, Alta Vista, Looksmart,
Direct Hit, Hotbot, AlltheWeb, The Open Directory and WebCrawler.
Here are the submission URL's for some of the top "crawler-based"
search engines. Click on the links below and fill out each of the
forms (look for the link that says "free submission").
Submission software is a much faster method that can be less accurate
because of the different submission guidelines that are constantly
changing. It's not one-size-fits-all, and no software program can
customize and update to the degree required for maintaining top
Paying for listings
The alternative to free submissions is to submit your site to what
are called "Paid Inclusion" programs which charge a fee to include
your site in their index. An advantage about paid inclusion programs
is that they list your site usually within a week. The cost varies
in pricing per URL for up to a year of inclusion in their listings.
Given this, it is highly recommended that any site owner establish
a search engine submission budget.
Types of Paid Listings
Banner Ads: All major search engines carry keyword-linked
banner advertising, either using graphical banners or text
Content Promotion: Many major search engines will
promote an advertiser's content, or their own content, on
their search results pages. This is usually done in a separate
area from the editorial results.
Paid Placement: Several major search engines carry
paid placement listings, where sites are guaranteed a high
ranking, usually in relation to desired words. The exact position
of these listings can vary. Sometimes, they appear above editorial
links. Other times, they appear at the bottom of editorial
content. "Sidebar" style runs alongside the search engine's
Paid Inclusion: This is where an advertiser might
be more deeply listed than other sites in the editorial results.
Unlike paid placement, this doesn't guarantee a particular
position in the main search results. However, also unlike
paid placement, it does interact directly with the editorial
results. Being more deeply listed can help an advertiser be
more likely to appear in response to a wide range of searches.
Paid Submission: This is where a search engine charges
to process a request to be included in its listings. Typically,
paid submission programs do not guarantee to list a site,
only to review and possibly include it in a faster time frame
than is normally done.
Pay-per-click search engines: With these search engines,
you literally pay a certain amount of money for every click
they send your way. You bid on keywords under which you want
to be searched. The more you bid, the higher you'll rank for
your search term and the more visible your link will be. The
largest and most popular is Overture, http://www.overture.com/.
A list of Pay Per Click programs is available from http://payperclicksearchengines.com/.
Not every page is indexed by Search Engines
Search engine databases and spiders are optimized to "read" HTML, the
basic language of the Web. Other types of programming languages contain
codes and format requirements that are incompatible with HTML. HTML can
carry links to these pages, but not full text of their content in their
There are also some types of pages that search engine companies exclude
by policy. It's a matter of selecting what and what not to include in
databases that are already huge, expensive to operate, and low revenue
Some types of web pages search engines choose to exclude:
Format of the page - are frequently not included in search engine
databases because surfers would infrequently or unsuccessfully search
Pages with images and no text are also often omitted because
without text, there are no keywords on which to search.
Pages formatted in PDF and other pages written using very
little if any HTML text.
Search engines also omit the contents of Flash, Shockwave,
and other programs like Word, WordPerfect, PowerPoint etc.
Pages requiring passwords to access them are also closed to
search engines, because spiders cannot type the password nor have
access to the password.