Robots.txt Tester
Paste a robots.txt and URLs, pick Googlebot, Bingbot, GPTBot or ClaudeBot, and see allowed or blocked plus the exact rule line (RFC 9309).
Still loading. If the tool doesn’t appear, reload this page.
Planned features and workflows
Upcoming additions related to this tool. Open a plan to see its scope. These features are coming soon.
- URL Evidence Inspector
Coming soon
About this tool
Robots.txt Tester (RFC 9309)
Test URLs against a robots.txt for Googlebot, Bingbot and AI crawlers like GPTBot and ClaudeBot, and see the exact line that allowed or blocked each one (RFC 9309).
A robots.txt tester tells you whether a crawler may fetch a URL under a given robots.txt, and which line decided it. This one implements RFC 9309: the crawler obeys the groups that name its product token (merged), or the * group if none do; the longest matching Allow or Disallow path wins; Allow wins a tie; * matches any characters and a final $ anchors the end of the URL. Paste the file and your URLs, pick Googlebot, Bingbot or an AI crawler such as GPTBot, ClaudeBot, PerplexityBot or CCBot, and every URL shows Allowed or Blocked with the matching line. Nothing is fetched or uploaded.
How to use Robots.txt Tester
- 1
Paste your robots.txt file into the robots.txt box; the counter under it shows how many User-agent groups, Sitemap lines and bytes were read.
- 2
Choose the crawler from the user-agent list — Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot and others — or pick Other and type any product token.
- 3
Enter the URLs or paths to check, one per line; each shows Allowed or Blocked with the winning Allow or Disallow line number, and Copy results copies the report as plain text.
Why use Robots.txt Tester
The line that decided it
Each URL shows the winning rule with its line number, its length in bytes, and every other rule that also matched, so a surprise block can be traced in seconds.
AI crawlers with sources
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot are listed with a link to each vendor's own crawler documentation.
Standard matching, no guessing
Group merging, longest match, the Allow tie-break, * and $ wildcards, percent-encoding normalisation and the 500 KiB parse limit all follow RFC 9309.
Common use cases
- Before deploying a robots.txt change, you paste the new file and your ten most important URLs to confirm none of them is blocked for Googlebot by a wildcard rule you did not expect.
- You want to stop AI training crawlers but keep AI search: you test GPTBot and OAI-SearchBot, or ClaudeBot and Claude-SearchBot, against the same file and see which groups each one actually reads.
- Search Console reports "Blocked by robots.txt" for a page, and you need to see the exact Disallow line, how long it is, and whether a longer Allow should have overridden it.
Frequently asked questions
How do I test my robots.txt now that Google retired the robots.txt Tester?
Paste the file and the URLs into a tester that applies the RFC 9309 rules, like this one. Google replaced Search Console's robots.txt Tester with a read-only robots.txt report in late 2023; that report shows which robots.txt files Google fetched and any parse errors, but it does not test individual URLs against a draft file.
Which rule wins when Allow and Disallow both match a URL?
The longest matching rule wins, measured in octets (bytes) of the rule's path after percent-encoding normalisation. If an Allow and a Disallow match with the same length, RFC 9309 says the Allow rule should be used. The order of the lines in the file does not matter.
Does Googlebot follow the User-agent: * group if there is also a Googlebot group?
No. A crawler obeys only the groups that name its own product token, and falls back to the * group only when no group names it. So rules under User-agent: * do not apply to Googlebot once a User-agent: Googlebot group exists; repeat any rules it should also obey.
What do * and $ mean in robots.txt?
* matches any sequence of characters, including none, and $ at the end of a path means the URL must end there. Disallow: /*.pdf$ blocks /files/guide.pdf but not /files/guide.pdf?download=1. To match a literal * or $ in a URL, write %2A or %24.
How do I block GPTBot or ClaudeBot but stay in search results?
Add a group for the training crawler, such as User-agent: GPTBot or User-agent: ClaudeBot followed by Disallow: /, and leave Googlebot and Bingbot under their own or the * group. OpenAI and Anthropic document separate tokens for their search and user-requested fetchers (OAI-SearchBot and ChatGPT-User; Claude-SearchBot and Claude-User), so blocking the training crawler alone does not block those.
Does robots.txt stop a page from being indexed?
No. robots.txt controls crawling, not indexing: a blocked URL can still be indexed from links without its content. To keep a page out of search results, let it be crawled and serve a noindex meta tag or X-Robots-Tag header instead.
Is Crawl-delay supported?
Crawl-delay is not part of RFC 9309, and Googlebot ignores it. Some crawlers, including Bingbot and ClaudeBot, document support for it. This tester lists it as a parser note and does not use it to allow or block URLs.