November 06, 2003

Field guide to search engine robots

Now that you know know a little bit more about robots.txt files, wouldn't you like to who is sending out the robots to your website in the first place?

Whenever a page is read from a web site, the log file records a number of details including the time, the IP address and usually the referrer page and the user agent. Some user agents are quite obvious, "Googlebot/2.1 (+", but others might just confuse you, "Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01". When you need to look up which bot is hitting your site to determine whether or not they should be added to your robots.txt file, take a look at: the search engine robots page.

This site will let you lookup user-agents, as well as providing you even more inforation about robots that will probably come in useful.

Posted by mark at November 6, 2003 11:32 AM | TrackBack
Post a comment

Remember personal info?