Developer tool
Robots.txt Generator
Build a conservative robots.txt file with sitemap and crawler controls.
Runs in your browserResult
How to use this robots.txt generator
- Enter the site hostname, choose the crawler policy, and check the sitemap URL in the generated robots.txt file.
- Review the generated options. Check syntax, escaping, and edge cases against the source before using the result in a live project.
- Check the output against the source, then copy or download the finished result.
About this robots.txt generator
Use the Robots.txt Generator to build a conservative robots.txt file with sitemap and crawler controls. Each control affects a specific part of the result.
You can also use it to change the controls to produce a different result.
Example
Enter example.com, allow normal crawling, and confirm that the generated sitemap line points to https://example.com/sitemap.xml.
Common uses
- → build a conservative robots.txt file with sitemap and crawler controls
- → change the controls to produce a different result
- → copy a useful result into the project it belongs to