How Search Engines Work

How Search Engines Work: Crawlers, Spiders and Bots Oh My!

Keeping track of billions of web pages isn't easy, yet today’s search engines continue to strive to serve up the most relevant search results no matter what you’re looking for. Have you ever wondered how they do it? If you want your website listed and well-positioned in the search engines, it’s important to understand the basics of how they work.

Crawlers, Spiders and Bots; Oh My!

When you enter a search term into a search engine, that search engine doesn't scan the entire Web right then and there to find your search results. That would take several weeks. What the search engines do is search their index or database of web pages to find those pages that are most relevant to your search term.

To build these indexes or databases of web pages, most of the popular search engines regularly deploy automated utilities called “crawlers, spiders or robots” to find web pages on the Internet. These crawlers “crawl” the Web seeking new and updated web pages to add to their search engine’s database.

Algorithm Who?

Now that you’re beginning to understand, here comes the term “algorithm.” An algorithm is nothing more than a program containing a particular set of rules. Each crawler-based search engine defines its own unique method of how web pages rank in its listings in a program called an “algorithm.” In order to protect themselves from competitors and those who wish to spam them, search engines do not reveal exactly how their algorithms work. That’s what makes optimizing websites for top rankings so much fun!

The Human Touch

Many crawler-based search engines give you the option of submitting web pages to their indexes manually as well. This is a great way for new websites that are virtually unknown to get indexed.

There are some search directories that do not use automated crawlers at all. Instead, their topic-based categories and ranking criteria is defined by human beings. Some search directories charge a fee to be included in their indexes. Pay for inclusion fees do not affect the placement of your website in search results, it only assures that your site will be listed in the index.

Play or Pay

OK, now let’s mix it up a little. You can also pay for specific placement with many of the search engines by purchasing ad listings for specific keywords or search terms. These ads are listed in the paid or sponsored area of the search engine results page.

The most popular type of pay-for-placement program is called pay-per-click. Pay-per-click programs typically allow advertisers to bid on keywords for a specified amount per click. You decide what you are willing to pay for each ad click-through to your site. The higher the bidder, the better ad position they receive. If you outbid your competition by a penny, you ad is listed above theirs on the search engine’s search results page. You pay only when a user clicks on your ad listing.

Achieve Your Online Potential

No search engine is like another—they all have their own set of rules and processes for obtaining and ranking website listings. Whether its crawlers and algorithms, human categorization, or bidding for ranking, it’s all about positioning your website where your prospects will see it. You can play the optimization game and create a site that is ‘search engine-friendly’, pay for search engine advertising or try your hand at both to reach top rankings. But one thing is for sure; if you don’t do either, your website is far from achieving its potential.

See also
SEO vs. SEM
SEM Glossary
SEM FAQ

top