The 4 Pillars of Natural Search- Pillar Number 1, Indexation

Posted by Sam Battin, Senior Search Specialist (Natural Search)

Before a Web site can hope to drive qualified natural search traffic from targeted keywords, it must be ensured that search engine spiders are able to index the most important pages of the site.  That’s why indexation is the first Pillar of Natural Search.  Indexation problems have to be cleared up at the start of any Natural Search Optimization campaign.  Search engine spiders generally encounter two main problems relating to indexation: (1) URLs with too many parameters (2) Design platforms that are difficult to crawl.

Complicated URLs

Search engine-friendly URLs are unique and generally consist of three parameters or less.  If a spider sees a URL that has multiple parameters and session IDs, it runs the risk of getting caught in a “spider trap” indexing duplicate content.   If each new visit to a page is associated with a unique session ID, a search engine spider can become trapped in an endless loop indexing pages that it has already crawled. This is because each time the crawler returns to a site and finds a page, whether it has already crawled that page or not, it gets assigned a new session ID and the URL changes. 

Search engine spiders are not likely to crawl and index URLs like this; if a URL contains too many “red flags” that indicate a dynamic URL, the spider may stop indexing that section of the site.

Flash / Java-Script

Search engine spiders don’t have fingers.  This means that unlike the users of your Web site, spiders can’t make selections from pull-down JavaScript menus or type information into JavaScript functions.  A JavaScript function, such as a pull-down menu that makes a user choose their state before going to the next page, is generally a wall that blocks a search engine spider.  This means that any content behind the wall will not get indexed unless the spider has another way to get to it.

While search engine spiders are getting better at indexing Flash pages, it’s still true that most Flash files are images and video, not the text that engines like Google and Yahoo! are now starting to read.  Many Flash-based sites also dynamically load graphics and text under the same URL, which causes indexation issues since URLs need to be unique.  When designing a site in Flash, natural search best practices must be followed in order to ensure that search engine spiders are able to index the pages.

Pillar Number 2: Content Optimization
Pillar Number 3: Link Building
Pillar Number 4: Distribution