Robots.txt Tester

Test whether Googlebot, Bingbot or AI crawlers such as GPTBot may crawl a page, read the live robots.txt and see exactly which rule decides.

Checked from our serverOnly what you enter is sentNo sign-up

What the Robots.txt Tester does

The Robots.txt Tester reads a site’s live robots.txt and tells you whether a crawler may fetch a page. Type the page’s address and, if you like, a crawler name: Googlebot is used unless you choose another. The tester follows the robots.txt standard that Google, Bing and others use: it picks the group for that crawler, or the group for all crawlers, then applies the longest matching rule, with Allow winning a tie, and * and $ in rules work as wildcards. It shows the deciding rule and its line, the same page for well-known search and AI crawlers, the sitemaps the file lists, and lines that crawlers ignore.

How to use it

  1. Paste the full address of the page to test, such as https://example.com/blog/post.
  2. Leave Googlebot as the crawler, or pick another, such as Bingbot or GPTBot.
  3. Press Test and read whether the page is allowed, and which line decides.
  4. Check the table to see the same page for other crawlers, and open the file to see the highlighted line.

Example

Testing a page on kitlantis.com shows it allowed for every crawler: its robots.txt is a single group that allows everything and lists the sitemap. Testing a news page on nytimes.com shows it allowed for Googlebot and Bingbot but blocked for GPTBot, ClaudeBot and other AI crawlers, each by its own “Disallow: /” group.

Common robots.txt mistakes

A leftover “Disallow: /” for all crawlers, often from a test site, hides the whole site from search. Rules are case-sensitive and match from the start of the path, so “Disallow: /Admin” does not block /admin. A crawler with its own group ignores the rules in the group for all crawlers, which surprises people who add a Googlebot group with only one line.

Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it, without a description. To keep a page out of search, allow crawling and add a noindex meta tag instead; Google stopped reading Noindex lines in robots.txt in 2019.

Questions people ask

What is sent to your server, and is it kept?

Only the address and crawler name you type are sent, so the Kitlantis server can read that site’s robots.txt. Nothing is saved with anything about you.

What if the site has no robots.txt?

If the server answers 404 or another 4xx code, crawlers treat every page as allowed. If it answers with a server error, Google pauses crawling until the file can be read again.

Can I block AI crawlers?

Many publish their names, such as GPTBot, ClaudeBot, CCBot and Google-Extended. Give each its own group with “Disallow: /”. Well-behaved crawlers follow it; robots.txt cannot force the others.

Why is a file over 500 KB a problem?

Google reads only the first 500 KB of a robots.txt and ignores the rest, so rules near the end may never apply.

Related tools