Robots.txt Generator

Make a robots.txt file: block folders, block AI crawlers like GPTBot, add your sitemap, and test any URL against Google’s rules.

Runs in your browserNo sign-upWorks offline
<meta name="description" content="Free tools that run in your browser.">
<meta property="og:title" content="Kitlantis">
<link rel="canonical" href="https://kitlantis.com/">

What the Robots.txt Generator does

The robots.txt Generator writes the file that tells search engines and other crawlers which parts of your site to stay out of. Start from a preset, list folders such as /admin/ and /cart to block, add exceptions, add your sitemap address and, if you wish, block 14 AI training and answer crawlers including GPTBot, ClaudeBot, Google-Extended and PerplexityBot. A built-in tester shows whether any crawler may visit any path, following Google’s matching rules, for your file or one you paste.

How to use it

  1. Choose a starting point: block a few folders, allow everything or block everything.
  2. List the paths to keep crawlers out of, one per line.
  3. Tick the AI crawler option if you want to opt out of AI training.
  4. Download robots.txt, upload it to the top of your site, and test URLs below.

Example

With the default rules, the tester shows Googlebot is blocked from /admin/login by “Disallow: /admin/”. With AI crawlers blocked, GPTBot is blocked from /blog/post by “Disallow: /” in its own group, while Googlebot can still crawl the blog.

How crawlers read the rules

Each crawler uses the most specific group that names it, falling back to User-agent: *. Within the group, the longest matching rule wins, and Allow wins a tie, so Disallow: /admin/ with Allow: /admin/public/ blocks the admin area but not the public part. * matches any characters and $ marks the end of a URL, as in /*.pdf$.

robots.txt is a request, not a lock: well-behaved crawlers obey it, but the file is public and blocked pages can still be found through links. Use a login to protect private pages, and noindex to keep a page out of search results.

Questions people ask

Is anything sent to a server?

No. The tool works entirely in your browser, so unpublished pages and private site details stay on your device.

Does blocking a page remove it from Google?

Not reliably. A blocked page can still be listed if other sites link to it. Use a noindex tag and let Google crawl the page to see it.

Does Google-Extended stop Google search?

No. It only opts you out of some Google AI training; Googlebot still crawls and ranks your site.

Where does the file go?

At the root of the site, so it opens at https://yoursite.com/robots.txt. Subfolders are ignored.

Related tools