Google’srolled out a new tool at Google Webmaster Central, a robots.txt generator. It’s designed to allow site owners to easily create a robots.txt file, one of the two main ways (along with the meta robots tag) to prevent search engines from indexing content. Robots.txt generators aren’t new. You can find many of them out there by searching. But this is the first time a major search engine has provided a generator tool of its own.
It’s nice to see the addition. Robots.txt files aren’t complicated to create. You can write them using a text editor such as notepad with just a few simple commands. But they can still be scary or hard for some site owners to contemplate.
More: continued here