The genisis of this article is that one of our clients had a well established website where they were generating dynamic pages in PHP which also generated a sitemap and submitted it to their Google Webmaster account upon change.  The sitemap contained well over 1000 URLs and was correctly read in Google Webmasters as having the correct number of submitted URLs, however Google Webmasters showed only 200 files in the Index for that sitemap.  This of course conflicted with any site search on Google Search which showed results of over 1000!  So Google Search knew about over 1000 pages, but was not attributing them to being the links in the sitemap, a definate head scratcher!

In examining their implementation it was determined that the client had taken a short cut in implenting their list of news articles.  Instead of printing out links to individual articles in the SEO friendly manner, their URLs were pointing to a non-SEO friendly page.  The links on their page pointed to:

http://www.example.com/news.php?article_id=2

instead of:

http://www.example.com/news/this-is-a-fantastic-article/

However, the sitemap contained the second version.

The problem with this is that Google has robots that crawl the site to find content in addtion to any sitemaps that are provided.  By using the first version on their site, Google searched the site and found the non-SEO friendly version.  They created duplicate content in the search engine’s eyes by having two urls that pointed to the exact same content and through the mystery that is Google’s back end processing, selected the unintended version.   What they really needed to do is be consistent and use the second version everywhere they have a link to that article, which is sometimes can be easier said than done.   You must ensure that your content always uses the SEO friendly URL (RSS feeds, backlinks, internal links, etc.) that way you will not get penalized for duplicate content.

In this instance we changed the URLs sitewide and made them all use the SEO friendly version and the client was able to increase their Indexed URL count from the sitemap to a number much closer to the number of URLs submitted in the Google Webmaster Tools.

In Part 2 of this article, we will implement a database driven news feed that generates and utilizes SEO friendly URLs for a website.

If you want to customize the error pages that are displayed for your site and you wrote the site from the ground up (NOT utilizing a Content Managment System such as WordPress, Drupal, Joomla, etc) creating the templates to display and some simple additions to your .htaccess file are all that are needed.

First, you will need to create a directory inside of your website for the templates to be stored such as ‘/errors’ which can be accessed at your website: http://www.example.com/errors.  Although this directory is not absolutely necessary, it will keep things cleaner in your site.

The major errors that you will want to handle are:

• 400 – Bad Request
• 401 – Authorization Required
• 403 – Forbidden
• 404 – Not Found
• 500 – Server Error

Create a webpage for each of these types inside your ‘errors’ directory and give them any name you prefer, hopefuly something meaningful such as:

• bad_request.html
• auth_required.html
• forbidden.html
• not_found.hmtl
• server_error.html

In each of the cases, you can create server side scripts that will be executed when the error is triggered as well.  If your normal webpages are setup to use server side scripts such as PHP then you can use PHP in your error handlers as well to email or log to a database when pages that don’t exist are requested for example.  They are simply web pages that will be displayed when the errors are triggered.

Once you have your files setup, the .htaccess magic happens.  You will add a line to your .htacces file for each of the pages to tell the Apache Web Server where to get the web pages to serve up when these errors are triggered.  The format for the .htaccess is:

ErrorDocument code response

For each of the errors we listed above with each of the pages created in the errors directory our additional lines in the .htaccess file will be:


ErrorDocument 400 /errors/bad_request.html
ErrorDocument 401 /errors/auth_required.hml
ErrorDocument 403 /errors/forbidden.html
ErrorDocument 404 /errors/not_found.html
ErrorDocument 500 /errors/server_error.html


That’s all there is to it! Simply create the html code to be displayed and point the Apache Web Server to the documents using the ErrorDocument directive in the .htaccess file.

For this example we have created handlers for the most common types of Apache Web Server responses that you would want to define. You do not need to generate any of these or you could add adddional handlers as your website demands. Read the full documentation on the Apache ErrorDirective.

Here is the case that we find most often: You install your blog on your main domain inside a sub directory so that your urls look like ‘http://www.example.com/blog’  and wrote some articles such as ‘http://www.example.com/blog/2011/11/this-is-how-you-do-it/’.  All is fine and dandy until you decide that you want to have a separate sub domain for your blog and relocate the root to ‘http://blog.example.com’ and your article looks like: ‘http://blog.example.com/2011/11/this-is-how-you-do-it/’.  All is well except for the people that bookmarked the old url will get a 404 error response when trying to access that old bookmark or backlink that no longer points to the article.  With a quick change of your .htaccess file, you can make the old urls point to the new ones.  Use this line in your .htaccess file in the blog directory:

RewriteCond %{HTTP_HOST} !^blog\.example\.com$
RewriteRule ^(.*)$ http://blog.example.com/$1 [R=301,L]

The RewriteCond is the conditions that must be met to allow the RewriteRule to take place and checks to see if the current URL does not already have the blog sub domain.   If it does not (the ! preceeding it) then the Rule can take place, which is rewrite anything in the original request to the new blog sub domain address.  Note that this is put into the .htaccess in the blog sub-directory so the rewrite engine will remove that before matching the string in the rule.  It also sends a 301 redirect (R=301) and does not process further (L).  These can be changed as you see fit based on the Apache mod-rewrite documentation.

We have incorporated this button on one of our client’s home page.  To see it in action visit http://www.fairwaysandroughs.com it is located near the top right of the page.

Three lines in the .htaccess file are required to have every page contain the host www. In order to utilize this method, you must be using a webserver with the ability to modify the running server using .htaccess files that are located in your webserver’s directory. In addition, the mod_rewrite module must be installed and activated on the server.

The following lines show the entries in your .htaccess file


RewriteEngine on
RewriteCond %{HTTP_HOST} ^example.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]


The first line tells the server to turn on the rewrite engine and allow RewriteCond and RewriteRule (among others) to be able to modify the request.

The second line defines a condition that must be met for the RewriteRule to take place. The condition is read in sentence format as “If the request is to the host ‘example.com’ without regard to case”. The %{HTTP_HOST} is a variable that the webserver sends to the mod_rewrite engine when a request comes in from a browser. if the user types in ‘http://example.com’ into the browser to visit the page, the variable %{HTTP_HOST} will have the value ‘example.com’. If the user types in ‘http://sub.docs.example.com/somepage.html’ then the %{HTTP_HOST} will have the value ‘sub.docs.example.com’. As you can see, the value of %{HTTP_HOST} will contain only the host portion of the request. The second part of the line is what the condition is equal to. This part is actually a pearl compatible regular expression (PCRE) (with modifications). The ‘^’ is a special character and marks the begining of the string to test and the ‘$’ marks the end of the string to test. In this example we want to make sure it is exactly equal to ‘example.com’ so the PCRE is exactly the string we want to searc for. The ‘[NC]‘ tells us to do the test without regard to case so the condition will be met if the user types ‘example.com’, ‘Example.com’, ‘EXAMPLE.COM’ or any other variation of lower or upper characters.

The third line is actually the one that does the rewriting. It will rewrite if the previous condition is set and the engine is on. There are three parts in this line after the RewriteRule and the first two are closely related. The first part is a PCRE search string for the query string (‘somepage.html’ for the example ‘http://www.example.com/somepage.html). Again this is a simple PCRE since we want to take it all we grab any character as many as there are enclosed between the start special character ‘^’, and the end special character ‘$’. The PCRE search pattern is simply (.*). When this matches it will be used in the latter portion as the value for the substitution pattern $1. The second part of the RewriteRule tells the webserver what the new request should be. In our case we want the request to be ‘http://www.example.com/ plus anything that was in the original request which is the data that was found appended to the previous request and contained in the substitution pattern $1. Appending these two together gives us our new request. The third part of the line [L,R=301] tells the mod_rewrite module to stop processing with the ‘L’ and send back the client a 301 Redirect.

The mod_rewrite engine is a powerfull tool to use in developing websites. It can also be very complex and difficult to troubleshoot. This is a simple example of using mod_rewrite, but the Appache documentation on mod_rewrite can be quite overwhelming. We felt an in-depth discussion on this simple concept would be valuable for the beginner web developer in getting started in mod_rewrite.

One of the main items to keep in mind when using cron jobs is that the executable that runs your cron jobs is a separate process than the webserver. This means that the “standard” webserver variables are not available to the cron script.

Using variables such as $_SERVER[SELF] are generated by the WEBSERVER, but a running cron job is run from the CRON DAEMON so these variables will not represent what you may be expecting from the webserver.

If your webserver supports .htaccess files, they are checked for for the configuration before the webserver serves up your page. You can make changes to the php configuration within the .htaccess files by setting the options there. Again, if you make changes to the php configuration within the WEBSERVER, they will not be available to the running cron job. You must explicitly make the changes witin the php file or from within a config file that is passed as a parameter to the php process that cron runs.

If you keep in mind that the cron daemon is separate than your webserver, you can avoid some easy to overlook common pitfalls of using cron jobs.

The main features of the AVR Logic Analyzer plug in are:

• Built for the Atmel™ AVR Studio 4 development system
• Fully integrated into the AVR Studio window management system for ease of layout
• Reads usable port configuration based on selected processor
• Stores up to 32,000 historical port values and Data Direction Register values for each port on the processor
• Displays a configurable number of visible port values for each selected port
• User can scroll back through historical port values to visualize value changes
• Quickly turn on or off the display of a single port (i.e. PB3) or a group of ports (i.e. PORTB)
• Allows the user to select different colors for a port state of on and off
• Display the port states in a different color if the port is setup to be an input as masked by the Data Direction Register (DDR)
• Saves user options in the project file

More information on the AVR Logic Analyzer plug in can be found at the plug in website at http://www.coynetechsystems.com/avrlogicanalyzer/index.php.
Coyne Technology Systems develops hardware, software, and embedded solutions and has over 15 years of experience in developing systems for the Personal Computer, internet and networked devices, handheld devices, and embedded systems. More information can be found on our website at http://www.coynetechsystems.com/index.php.

A sitemap file is an XML file that defines the pages that you provide to search engines that you specify the files that are to be indexed in your site.  By using this protocol, you specify the exact pages that want the search engines to find, even though they may be database driven, or difficult to find through deep links on your site.  Sitemap files can be indexes where they contain a list of other sitemap files or sitemap entries containing links to web pages.  More information on the sitemaps protocol can be found at sitemaps.org.

This add on reads the knowledge base categories and articles from the WHMCS database tables and generates a list of the SEO titles from those table entries in a sitemap entry file.  It then reads the specified sitemap file, if it exists, and matches up the database table entries with the entries in the sitemap file.  If there are KB categories or articles that are not included in the sitemap file, the WHMCS admin is given the ability to add the articles to the sitemap file.  If the categories and articles are already in the sitemap file, then the WHMCS admin has the ability to update the timestamp in the sitemap file or remove the item from the sitemap file entirely.  The sitemap file can also contain additional static entries that will be listed in a separate table at the bottom of the page.   The WHMCS admin then has the ability to remove or update the timestamps on the static items.

The problem with creating sitemap entries from the WHMCS database is that the KB articles and categories are generated from the database versus stored as files in the file system which would contain timestamps of when the file was last written.  The WHMCS database does not store the time the article or category is updated so there is no way to know when the last update to the article was made.  To overcome this in the add on, the WHMCS admin must manually specify when to update the sitemap lastmod time of the articles by checking the box in the update column and re-generating the sitemap file.

To access this WHMCS module, the admin logs in and selects the Addon Modules menu item under the Utilities Menu.  From there, they are presented with a list of add on modules available.  If the KB Sitemap Generator was installed correctly, the module is listed as Kb Sitemap Gen.  Clicking on this title will access the module, which first must be installed.  To install, select the button that says “Install”.  This will create an additional WHMCS database table which will hold the sitemap file name.

Once the add on module is installed, the module presents three general areas of information. 1) WHMCS and Sitemap File information, 2) Table to manage WHMCS KB database data in the sitemap file, and 3) Table to manage static sitemap tables.

The WHMCS and Sitemap File information shows the WHMCS Root Path which is the path WHMCS believes it is installed in.  This URL path will be used to generate the URL in the sitemap file.  The next entry in this area shows the file path that the sitemap file will be created or exists in.  It defaults to the path where WHMCS is installed plus the file name “kbsitemap.xml”.  The file name and location can be changed by entering optional additional path information and file name and pressing the “Set File” button.  This will cause the add on module to store the new file location in its database table and will be available for future use of the module.  The input box will accept additional path information as well, so the sitemap file could reside in a different location.  For example setting the file to ../kbsitemap.xml will generate a sitemap file one directory higher than the WHMCS install directory.

The “Files From WHMCS KB Database” table lists the Type, URL , LastMod, and three check boxes to Add, Update, or Remove the item.  The Type is whether the item is a Category or an Article and each type is displayed with a different color in the table.  The URL is the address that is generated from the WHMCS installed directory plus the knowledge base sub-directory, plus the category or article id from the database, and finally the SEO friendly filename generated from the title or name of the article or category.  The LastMod column is the LastMod time from the sitemap if the item is in the sitemap, otherwise it is empty if it has not been generated.  The three check box columns allow the admin to manage the items in the sitemap file and all changes are made when the user presses the “Make Changes To Sitemap” button at the bottom of the admin module.  If the item is already in the sitemap file, the Add check box will be disabled and if the item is not yet in the sitemap file, the Update and Remove check boxes will be disabled.

The following image shows the WHMCS and Sitemap File Information as well as the begining of the table that lists the Files From WHMCS KB database.

KB Sitemap Generator Top Half

In addition to the sitemap information generated from the WHMCS database, this add on module can also manage additional static sitemap file information.  Beneath the Files From WHMCS KB Database area, the add on module will display sitemap entries that are not generated from the WHMCS knowledge base database tables.  These could include static entries or KB article and category that were manually entered and contained spelling or format errors.  The image below shows several entries that were static entries for the affiliates, tutorials, downloads, and announcements pages.  From this table the admin can Update or Remove the items in the sitemap file.

The image below shows the bottom of the WHMCS KB database table items as well as the static sitemap items.

KB Sitemap Generator Bottom Half

Now you can add your automatically generated sitemap to a sitemap index file.  For more information on using sitemaps and sitemap index files, click on the link to the article ”Building A Comprehensive Sitemap for SEO” in our blog.

This add on was developed for WHMCS version 4.1.1.   For more information on WHMCS, visit their home page.

You can download the script as a zip file for free in our shop by following this link.  Installation instructions can be found in the text file contained in the zip file.

The main features of TailWin log file viewer are:

• Built for the Windows™ Operating System
• Continuously monitors a text file for changes and displays the new lines in the file
• Shows new text lines in reverse order to quickly see changes
• Saves most recently used files for ease of opening
• Ideally suited for log file viewing such as webserver error logs and webserver access logs

More information on TailWin log file viewer can be found at the plug in website at http://www.coynetechsystems.com/tailwin/index.php.
Coyne Technology Systems develops hardware, software, and embedded solutions and has over 15 years of experience in developing systems for the Personal Computer, internet and networked devices, handheld devices, and embedded systems. More information can be found on our website at http://www.coynetechsystems.com/index.php.