How to use the Robots.txt Generator
What it does
The Robots.txt Generator builds a valid robots.txt file that tells search engine crawlers and AI bots which parts of your site they may or may not access. It helps you allow the crawlers you want, block the paths you want to keep private, and point crawlers to your XML sitemap, all without memorizing the syntax.
How to use it
- Open Robots.txt Generator under Tools.
Reading the results
The preview shows the finished directives grouped by user agent. Each Disallow line blocks a path, and each Allow line explicitly permits one. Remember that robots.txt controls crawling, not indexing, so do not rely on it to hide sensitive pages. Double-check that you have not accidentally blocked your whole site with Disallow: /. If you want granular control over AI crawlers, use this alongside the Crawler Governance features.
Credits
The Robots.txt Generator is free and costs 0 credits. Tip: always include your sitemap line and test the file in your search console before and after uploading.
Was this helpful?