Understanding the difference between the robots.txt file and Robots <META> Tag is critical for search engine optimization and security. It can have a profound impact on the privacy of your website and customers as well. The first thing to know is what robots.txt files and Robots…

Please visit Search Engine Land for the full article.

This article is from: 

Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?