What the Robots.txt Generator does
The robots.txt Generator writes the file that tells search engines and other crawlers which parts of your site to stay out of. Start from a preset, list folders such as /admin/ and /cart to block, add exceptions, add your sitemap address and, if you wish, block 14 AI training and answer crawlers including GPTBot, ClaudeBot, Google-Extended and PerplexityBot. A built-in tester shows whether any crawler may visit any path, following Google’s matching rules, for your file or one you paste.
How to use it
- Choose a starting point: block a few folders, allow everything or block everything.
- List the paths to keep crawlers out of, one per line.
- Tick the AI crawler option if you want to opt out of AI training.
- Download robots.txt, upload it to the top of your site, and test URLs below.
Example
With the default rules, the tester shows Googlebot is blocked from /admin/login by “Disallow: /admin/”. With AI crawlers blocked, GPTBot is blocked from /blog/post by “Disallow: /” in its own group, while Googlebot can still crawl the blog.
How crawlers read the rules
Each crawler uses the most specific group that names it, falling back to User-agent: *. Within the group, the longest matching rule wins, and Allow wins a tie, so Disallow: /admin/ with Allow: /admin/public/ blocks the admin area but not the public part. * matches any characters and $ marks the end of a URL, as in /*.pdf$.
robots.txt is a request, not a lock: well-behaved crawlers obey it, but the file is public and blocked pages can still be found through links. Use a login to protect private pages, and noindex to keep a page out of search results.
Questions people ask
Is anything sent to a server?
No. The tool works entirely in your browser, so unpublished pages and private site details stay on your device.
Does blocking a page remove it from Google?
Not reliably. A blocked page can still be listed if other sites link to it. Use a noindex tag and let Google crawl the page to see it.
Does Google-Extended stop Google search?
No. It only opts you out of some Google AI training; Googlebot still crawls and ranks your site.
Where does the file go?
At the root of the site, so it opens at https://yoursite.com/robots.txt. Subfolders are ignored.