The Importance of Crawlable Link Structures

It is vital that webmasters and company owners invest time and effort in creating superior web optimization for their sites. What often happens is that webmasters create poorly devised web structures, and only a few pages end up being searched and indexed. The fewer pages that are crawled, the less important your website is—in fact, it may even prevent your website from being found for months on end!

The key to effective website optimization is to make sure you have good navigation and a series of links that make it easy for users and spiders to read the entire website. This can be done by text based navigation, anchor texting, or even image navigation (provided you add alt attributes to each image). On the other hand, links that rely on Java or Flash technologies are not easy to crawl, and this will affect your search traffic. At the very least, it is highly recommended that you build an alternative HTML page to compensate for your search-dead Flash page.

Second, it’s important to add an XML Sitemap to your site so that you can direct crawlers where to internally see the entire site. Consider this XML document a “blueprint” for searching and indexing your site. Be sure to add plenty of links to and from your pages, internally and externally, so that you can alert crawlers searching other websites to your new content. Don’t underestimate in-content links, as they tend to have high CTR and can add special significance to links because of anchor text. In fact, in the thesis The Anatomy of a Large-Scale Hypertextual Web Search Engine, the computer science department states that anchor text is advantageous because they can include “more accurate descriptions of web pages,” and “can exist for documents which cannot be indexed by a text-based search engine.”

Failure to meet these standards could result in dozens of “orphan pages”, which are pages that have few or no links coming in from any other pages. That’s right, few links on the page are just as risky as having no links on the page. Internal and external linking works well, but always make sure that linking pages are suitable and related pages to your main issue. Websites that have “buried” pages will often find themselves very low on the overall page rank, in addition to the orphan pages themselves being completely snubbed.

Last but not least, familiarize yourself with the Robot.txt file, which is an HTML file that essentially orders crawlers what to do. Remember to check your robot.txt file to make sure that all of your pages are up to date, and still searchable by crawlers.

Efficiently designed crawlable link structures will determine whether your site is search friendly or out in the middle of nowhere!