LinkPopularity.com: The Free Link Popularity Service is a great resource for anyone who wonders, "why is that site ranked so high, and my site ranked so low."
This service will quickly help you find out how many other URLS link to the URL that you submitted. It will let you know this for Altavista, Google and Hotbot.
| See who links to your web site. |
Now that you know know a little bit more about robots.txt files, wouldn't you like to who is sending out the robots to your website in the first place?
Whenever a page is read from a web site, the log file records a number of details including the time, the IP address and usually the referrer page and the user agent. Some user agents are quite obvious, "Googlebot/2.1 (+http://www.googlebot.com/bot.html)", but others might just confuse you, "Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01". When you need to look up which bot is hitting your site to determine whether or not they should be added to your robots.txt file, take a look at: the search engine robots page.
This site will let you lookup user-agents, as well as providing you even more inforation about robots that will probably come in useful.
If you ever take the time to look in your web server access logs, you might be surprised to see that you are getting tons of 404 errors for a file called robots.txt. While you could spend the time to make an elaborate file, why not spend 30 seconds to make a very simple one that will reduce the number of 404 errors, and make search engine spiders that much happier to visit your site.
A generic robots.txt file that welcomes all robots and denies none would look like this:
User-agent: *
Disallow:
More examples can be found at clockwatchers.com.
If you ever take the time to look in your web server access logs, you might be surprised to see that you are getting tons of 404 errors for a file called robots.txt. While you could spend the time to make an elaborate file, why not spend 30 seconds to make a very simple one that will reduce the number of 404 errors, and make search engine spiders that much happier to visit your site.
A generic robots.txt file that welcomes all robots and denies none would look like this:
User-agent: *
Disallow:
More examples can be found at clockwatchers.com.
People are always asking me, "Hey Mark, how do I get my website listed/ranked higher in search engines." I'm never sure what to tell them, because there are so many variables involved. Came across this great guide, 26 steps to 15k a Day on searchengineworld.com. If you were to follow it, you would be well on your way to a better site.