Bots, crawlers, spiders? What do they all mean?

09 July 2007

If you trying to analyse your website statistics, you're probably thinking what are all these nasty creatures doing crawling all over the internet and will they terrorise my website?

Never fear, many of these creatures are actually the ones that you wish would come to your site more often, and you would definitely would not want to stamp them out. Let's find out why.

"Bots", "crawlers" and "spiders" are terminology which mainly relates directly to search engines. These computer programs allow leading search engines such as Google to index billions of web pages, with more and more added all the time. It is an automated process where bots, spiders and crawlers typically visit your website on a daily or monthly basis.

So what makes these creatures to important?
As we all know the higher your website ranks within search engine results, the more hits your website will receive. The primary role of spiders, bots and crawlers is to rummage around your website and gather as much information as possible about your website.
By allowing these crawlers to easily access your website and relevant pages will gain points and the more points you accumulate means higher your website ranks in search engine rankings and more people will find out about your company.

Leading search engines such as Google, MSN and Yahoo has specialised bots which are sent out to search for particular information within your website. Such as Google has a number of crawlers, Googlebot crawlers your website, Imagebot is their spider crawler which searches for images, and even Google-bot-Mobile for mobile phones.

So if you want your website to rank higher in search engine results, let's look at how do you get the spiders, crawlers and bots on your side.

Firstly, you must ensure your website is search engine friendly. As spiders only read HTML (hyper text mark up language), which is what the site is constructed. At Datalink ensure all our website are W3C compliant, which means they fulfil the requirements for valid HTML therefore enabling crawlers to go through your website. With the use of Freestyler CMS we ensure that this process is made very easy, as spiders only read text and cannot read JavaScript or Flash at all, so fancy animations on website pages are pretty much worthless from a search engine point of view.

In addition, bots cannot see pictures so it pays to make sure all images have ALT tags to help the crawlers understand and index the content.

So what can I do now?
Within Freestyler there are a number of fields which enable you to fill a description for each image and page to ensure your website is as search engine friendly as possible.

Although the look and experience of a site is important to visitors, to ensure you have repeat visits from these crawlers you must ensure your website is as accessible as possible. This can easily be done within Freestyler by following the four tips below:

1. Images: For each image we provide an image description field, this enable administrates to directly populate the ALT tag fields. The description should be short a simple and to the point. ALT tags not only help search engine but also ensure readers such as JAWS to read the information within your website.
2. Friendly URL: it is also very important to fill in the friendly URL with a meaningful name; this provides another means for allowing search engines to index your page and also gives meaning and value to the page created.
3. Meta Data: This is probably a tab which you barely use, however there are a number of fields with this tab which can enable you to enhance the ranking of pages over others. For each page within Freestyler you can enter in keywords and descriptions. This will automatically populate within the HTML code
4. Title and Summary Fields: Although these fields may seem minor, they can severally impact the way in which your website ranks. Within each of these fields the main keywords should be place as search engine crawlers give these fields more propter over the body text of the page.

Finally, once you have done all this, submit your site to the search engines, and ensure that at least one website links to your site. This is something that we do for every customer's site that we develop.

In the huge World Wide Web, to ensure that your site doesn't get lost and that you receive as much exposure and rank highly in search engines it is imperative that you keep search engines happy, by doing the basics.