Spider Session Remover

for osCommerce Online Merchant v2.2

Spider Session Remover v1.0 (Jan 15th 2005) ================================== This is the official release of the Spider Session Remover. This contribution uses Apache mod_rewite to look for specific spiders, and remove the session (osCsid) from the URL, and return a '301' back to the spider. Basically, if the spider tries to do this: GET /www.example.com/product_info.php?products_id=24&osCsid=ac8d8926059625ecb8dd9115f91d5f8a the Apache mod_rewrite will rewrite the url to be: GET /www.example.com/product_info.php?products_id=24 and also return a "301" (Moved Permanently) to the spider. The problem ========= You may use one of the following: * 2-2MS2 "Prevent Spider Sessions" admin feature is set to true. * SID Killer contribution (http://www.oscommerce.com/community/contributions,952) * Spider Killer for MS1 contribution (http://www.oscommerce.com/community/contributions,1089) All of these features are very good, and aim to prevent spiders from adding an session ID (osCsid) to the url. However, what if a spider started to crawl your website BEFORE you enabled one of the above features ? What can happen, is that the (previously) harvested URLS with SIDs in them will show as results in search engines. Afterwards, often many months later, you will still see the spider trying to access the the URLs it harvested earlier with the session ID in it. In summary, URL's with sessions ID's were harvested PRIOR to any session disabling, and therefore these URL's are now indexed in search engines, and the spiders continue to re-visit your website using the URL's with the 'osCsid' in them. The Solution ========= So, how do we remove these session ID's for the spiders that continue to use the previously harvested URL ? By the use of Apache mod_rewrite, look for the spider agent name, and if the condition is true, re-write the URL without the 'osCsid' in it, and ALSO return a "301" back to the spider. What results from the mod_rewite ======================== As to what the search engines will do, they'll see the 301-Moved Permanently response, re-fetch the page from the new (osCsid-less) URL given in that response, and, ...... after a while, update their database to use the new URL. Credits ====== Being no Apache mod_rewrite guru, I would not have been able to do this on my own, so special thanks to "jdMorgan" (Jim) from the WebmasterWorld forums (http://www.webmasterworld.com) who helped me a lot with getting the mod_rewite code correct. Legal Stuff =========== This contribution is released under GPL (see the licence file included in the archive download). =============================