26 Mar


Designing WEBSITE Structure for Spiders

This article was written in January 2010, for more pertinent information on Search Marketing today, please send us an email.


In Phase 1 we learned about structuring a website with the site visitor in mind.  Here in Phase 2:  Designing the Structure for Spiders we will learn how to develop an effective website structure to work with website spiders or crawlers or bots.


At this stage of search engine optimization, SEO, we will explore some of the aspects of home page navigation, global navigation, using sitemaps and crawl pages.

Internal Linking: the Basics

In Chapter 5:  Keyword Strategy Development you brainstormed keywords and phrases and then used some internet tools to create clusters to use to improve page ranking of specific target pages.


An important aspect of developing effective internal linking is to focus as much of your website’s PageRank onto these pages you have targeted as “high priority”.  The amount of page rank  a page can give to other pages in the website is limited to the amount of ranking it picks up from inbound links (links coming to that page from other websites).  The in turn, the amount of PageRank it has to give to other pages is divided between all the links that go out to other pages.


Therefore, it becomes very important to decrease the number of outgoing links to less important pages, while at the same time increasing links to pages that are considered to be vital to the website’s success.

Home Page Navigation

The homepage of the website is uniquely important in that much of the PageRank of the website flows both into and out of that page.  For this reason, the following steps are vital when considering a linking strategy from the home page:


  • Remove any unnecessary links to overhead pages (pages such as Terms of Use, Privacy Policy, contact forms, etc.).  If you feel you need to link to such pages, it is easy enough to combine them all into one single page allowing you to only use one link from the homepage to get to all that information.


  • Avoid linking to other websites from your homepage.


  • If you use a site map link to it from your homepage to help them get indexed.

Global Navigation

Global Navigation is defined as links to a website’s level 2 pages (remember Category Pages from the last lesson?) and which occur on every page of the website.   Typically those are the main links listed near the top of a website, or along the left panel of a site.


If your website uses global navigation elements it is important to reduce the number of unnecessary links going out from interior pages of the website, just like on the homepage.

Using Site Maps

A site map, or sitemap, is a page that contains an overview of the pages on a website.  For smaller websites, it might list every page on the site while on larger sites it might only list major categories.  For our purposes, a sitemap also contains outbound links to those pages.


The site map should be linked from the homepage and should link to at least all your second and third level pages (pages that link from your home page and from the pages that those link to).  You need to be sure not to link to more than 150 pages from the site map page, as web spiders or crawlers will not typically go beyond 150 outbound links from any page.  (If for some reason you have more than 150 outbound links to include in your sitemap, just add a second or third page to the sitemap).


Typically, site maps are not used by site visitors but are still useful to you since they can funnel some page rank to important pages on your site.


Want to know more about Search Engine Marketing People?

Search Engine Marketing People can help!

Multilingual Search Marketing, Localised Web Presence & Customer Services

Contact SEMP today if you want to improve your Search Marketing ROI.

Share this

Leave a reply