Sitemaps

In the beginning was the HTML sitemap. This was a simple web page, generally listed at the bottom of the index page, and listed all the pages of your site. It was supposed to be an easy summary of all the pages on your site and how to get there, although surely if you had nice simple one or two level navigation, you really shouldn't need to have one. It was also a good form of backup, in case you used fancy Javascript navigation, to make it easy for search engine spiders to find all pages of your site, within two clicks of the index page.

Then came Google Sitemaps. These were written in XML, and designed to help Google find obscure pages of your website more easily, telling Google how important they were, how frequently updated, and to notify Google of new pages. Theoretically it was particularly good for dynamic database changes that Google might find hard to detect. There are happy case studies at the Google Sitemap help page.

A side benefit for Google is that you need to have a Google account, so they have more information about you. But they do make it worth while, by giving you some nifty statistics on how and where visitors found your website through Google search and whether you had any link errors on your site.

Then Yahoo jumped on the bandwagon. Their submit page allows you to provide either a simple text list of pages, or a zipped file, or an XML file. You could even submit your Google sitemap file to save yourself effort.

Webmasters seem to be in two minds about Google's XML sitemaps, whether it is giving too much information to the search engines, and whether they have seen any real indexing benefit yet. But if you've got nothing to hide, and have a quality fresh site, then it simple can't hurt, and should help.

Google do provide a sitemap generator, but it is a Python-based tool, so most people go for a free generator, such as the AuditMyPC Free Sitemap Generator. Then submit it at Google and Yahoo and start checking on your indexing.