Robots.txt Generator
Generate a robots.txt file for your website
Robots.txt Settings
Select which crawler the rules apply to.
Optional XML sitemap URL.
Support for crawl-delay varies by crawler.
Generated robots.txt
Important:
A robots.txt file gives crawlers instructions about which URLs they may crawl. It is not a security mechanism and should not be used to protect private or sensitive content.
A robots.txt file gives crawlers instructions about which URLs they may crawl. It is not a security mechanism and should not be used to protect private or sensitive content.