The robot text file, better known as robots.txt, is a long-running Web standard which helps prevent Google and other search engines from accessing parts of your site. Why would you want to block ...

output-file robots.txt Where to write the resulting robots.txt file input-file base-robots.txt An existing robots.txt. Will be added to the top of the output file. Must not be the same as the ...