When robots (like the Googlebot) crawl your site, they begin by requesting
http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within WordPress (using
Options -> Robots.txt).
Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the WordPress admin menu.
This plugin should work with most versions of WordPress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.
Note that robots make only top-level requests for robots.txt files. If you have WordPress installed in a subdomain (e.g.
http://blog.example.com/) or in your root (e.g.
http://example.com/), this plugin will work as intended. But if you have WordPress installed in a subdirectory (e.g.
http://example.com/blog/), then this plugin won’t do much for you, since the search engines won’t look for
http://example.com/blog/robots.txt, only for
Also, requires that you be using some form of permalink structure. If links to blog posts look like
http://example.com/index.php?p=36, this won’t work.
If you post your support questions as comments below, I probably won’t see them. If the FAQs don’t answer your questions, you can post support questions at the KB Robots.txt plugin page on my site.
Free Download: Download this wp-plugin