The genisis of this article is that one of our clients had a well established website where they were generating dynamic pages in PHP which also generated a sitemap and submitted it to their Google Webmaster account upon change.  The sitemap contained well over 1000 URLs and was correctly read in Google Webmasters as having the correct number of submitted URLs, however Google Webmasters showed only 200 files in the Index for that sitemap.  This of course conflicted with any site search on Google Search which showed results of over 1000!  So Google Search knew about over 1000 pages, but was not attributing them to being the links in the sitemap, a definate head scratcher!

In examining their implementation it was determined that the client had taken a short cut in implenting their list of news articles.  Instead of printing out links to individual articles in the SEO friendly manner, their URLs were pointing to a non-SEO friendly page.  The links on their page pointed to:

http://www.example.com/news.php?article_id=2

instead of:

http://www.example.com/news/this-is-a-fantastic-article/

However, the sitemap contained the second version.

The problem with this is that Google has robots that crawl the site to find content in addtion to any sitemaps that are provided.  By using the first version on their site, Google searched the site and found the non-SEO friendly version.  They created duplicate content in the search engine’s eyes by having two urls that pointed to the exact same content and through the mystery that is Google’s back end processing, selected the unintended version.   What they really needed to do is be consistent and use the second version everywhere they have a link to that article, which is sometimes can be easier said than done.   You must ensure that your content always uses the SEO friendly URL (RSS feeds, backlinks, internal links, etc.) that way you will not get penalized for duplicate content.

In this instance we changed the URLs sitewide and made them all use the SEO friendly version and the client was able to increase their Indexed URL count from the sitemap to a number much closer to the number of URLs submitted in the Google Webmaster Tools.

In Part 2 of this article, we will implement a database driven news feed that generates and utilizes SEO friendly URLs for a website.

One of the main items to keep in mind when using cron jobs is that the executable that runs your cron jobs is a separate process than the webserver. This means that the “standard” webserver variables are not available to the cron script.

Using variables such as $_SERVER[SELF] are generated by the WEBSERVER, but a running cron job is run from the CRON DAEMON so these variables will not represent what you may be expecting from the webserver.

If your webserver supports .htaccess files, they are checked for for the configuration before the webserver serves up your page. You can make changes to the php configuration within the .htaccess files by setting the options there. Again, if you make changes to the php configuration within the WEBSERVER, they will not be available to the running cron job. You must explicitly make the changes witin the php file or from within a config file that is passed as a parameter to the php process that cron runs.

If you keep in mind that the cron daemon is separate than your webserver, you can avoid some easy to overlook common pitfalls of using cron jobs.

We are constantly being asked to develop more and more complex sites. Those that include static pages, blogs, shopping carts, and more. The advancement of software these days is allowing each of them to create XML sitemap files to define the individual pages of their particular specialty. WordPress blog software can be loaded with a plug-in to allow it to generate a XML sitemap for example. If your install of that software is in a subdirectory say /blog/ then it will generate it in that directory. So a site with a Zen Cart installation in the /store/ directory can have another sitemap in that directory for its produts. Using a sitemap index file you can easily allow each piece of software to generate it’s sitemap and pull them all together into a single file for the search engines to find.

The first thing to do is list out all software that generates a sitemap and where it resides. In the previous examples lets say we also have some static pages, so what we will have is a sitemap at /blog/sitemap.xml, a sitemap located at /store/sitemap.xml and a root sitemap at /sitemap.xml. The first thing to do is rename the /sitemap.xml to something say /sitemap_static.xml.

Now create a new text file called /sitemap.xml. This file will also be XML in the same manner as the sitemaps, but it will list each of the other sitemaps. The first line in the file will be the tag to indicate that it is XML and its encoding:

<?xml version="1.0" encoding="UTF-8"?>

The next line states that it is a sitemap index and the particular schema it adheres to:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

Now add a sitemap entry for each sub-sitemap that you have giving it a location and a last modified time:

<sitemap><loc>http://www.example.net/sitemap_static.xml</loc><lastmod>2009-11-12</lastmod></sitemap>
<sitemap><loc>http://www.example.net/blog/sitemap.xml</loc><lastmod>2009-11-10</lastmod></sitemap>
<sitemap><loc>http://www.example.net/store/sitemap.xml</loc><astmod>2009-11-15</lastmod></sitemap>

Don’t forget to close out your sitemap index tags:

</sitemapindex>

Now when you make chages to only a portion of your site, you can either submit only the sitemap of the portion that changed to the search engines or the index file and let them determine what has changed. So if you write a new blog entry, and your /blog/sitemap.xml file is updated, you can either resubmit just the /blog/sitemap.xml or /sitemap.xml. Don’t forget to change the lastmod time in your index file when changes are made to the sub-sitemap files.

Put a line in your robots.txt to point to your sitemap index file. This will allow search engine spiders to find the files in your site. You may have submitted your site to a number of search engines, but that one that you left out may still eventually find your site and when it does, the first thing it will look at is the robots.txt file and see your sitemap index. The format for the robots.txt file to include a sitemap is:

Sitemap: http://www.example.com/sitemap.xml

This will tell all the search engines where your index file is and once they read it, they will find that it is an index and pull in all the sub sitemaps making your site much more search engine optimized (SEO)!

For more information on sitemaps, see http://www.sitemaps.org