Now that you know know a little bit more about robots.txt files, wouldn't you like to who is sending out the robots to your website in the first place?
Whenever a page is read from a web site, the log file records a number of details including the time, the IP address and usually the referrer page and the user agent. Some user agents are quite obvious, "Googlebot/2.1 (+http://www.googlebot.com/bot.html)", but others might just confuse you, "Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01". When you need to look up which bot is hitting your site to determine whether or not they should be added to your robots.txt file, take a look at: the search engine robots page.
This site will let you lookup user-agents, as well as providing you even more inforation about robots that will probably come in useful.
Posted by mark at November 6, 2003 11:32 AM | TrackBack