Free tools Robots.txt Tester
Robots.txt tester,
for search and AI crawlers.
Check whether a crawler may fetch a page, see the line in robots.txt that decides it, and find out at a glance which AI assistants can read your site.
How robots.txt decides
robots.txt is a plain text file at the root of a site that tells crawlers which addresses they may
fetch. Each crawler looks for a group that names it, and falls back to the User-agent: *
group when none does. A group that names a crawler replaces the general one entirely.
Inside the group, the most specific matching rule wins, measured by the length of its path. When an
Allow and a Disallow are equally specific, Allow wins. * matches anything and $
marks the end of the address. This tester follows Google's rules exactly, and shows the line that won.
Why AI crawlers are listed separately
AI companies run two kinds of crawler: one that gathers training data, and one that fetches a page live to answer a question. Blocking the first is a choice about how your content is used. Blocking the second keeps your pages out of answers in ChatGPT, Claude and Perplexity that could have cited you.
One thing robots.txt does not do: keep a page out of search results. A blocked page can still be listed if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex tag.
Questions
Where is my robots.txt?
Always at the root of the domain: https://example.com/robots.txt. A file in a folder is ignored, and each subdomain needs its own.
Does Disallow stop a page appearing in Google?
No. It stops Google reading the page, but the address can still be listed if other pages link to it. Use a noindex tag, on a page crawlers can reach, to keep it out.
What happens if robots.txt is missing?
Nothing bad: a missing file means every crawler may fetch every page. An error from the server is different, and Google stops crawling until it answers normally.
Should I block GPTBot and other AI crawlers?
It depends on what you want. Blocking training crawlers keeps your content out of future models; blocking the live ones, like OAI-SearchBot and ChatGPT-User, also keeps you out of the answers they give today.
Can I test a change before publishing it?
Yes. Open "Test your own robots.txt", paste the new file, and test any address against it. Nothing is changed on your site.
Every check, on one page, ranked by what matters.
These tools each answer one question. A WebRankPage report answers all of them at once for a page, and more than fifty others, with the evidence from your own source and the fix for each.