Validate and Test Your Robots.txt
A single typo in robots.txt can block search engines from your most important pages, or let them into areas you meant to keep private. Paste your file above and it's checked line by line as you type. Each problem comes with a plain-English explanation, and you can test any URL to see whether Googlebot, Bingbot, AI crawlers or any other bot is allowed to crawl it, and which rule decided. Everything runs in your browser.
How to check a robots.txt file
- Paste your robots.txt into the left box, open the file, or click From a site and enter your domain to load the live file.
- Read the Issues list. Errors are lines crawlers will ignore, warnings are rules that probably don't do what you want, and tips are optional improvements. Click a line number to jump to it.
- Type a path or full URL into the tester and pick a crawler. You'll see Allowed or Blocked and the exact rule that decided it.
- Open Rules by crawler to see every group in plain terms: who it applies to, what's blocked and what's allowed.
What the validator checks
- Typos crawlers ignore:
Dissallow,Useragentand missing colons, with the fix suggested. - Rules that can't work: Allow or Disallow before any User-agent line, full URLs where a path belongs, paths without a leading
/, and a$in the middle of a rule. - Costly mistakes:
Disallow: /blocking your whole site from every crawler or from Google or Bing, and rules that block CSS and JavaScript files Google needs to render your pages. - Unsupported directives:
Noindex(Google dropped support in 2019),Crawl-delay(ignored by Google), and Yandex-only directives likeHostandClean-param. - Sitemaps: relative sitemap URLs, duplicates, or no sitemap at all.
- Structure: the same crawler in more than one group, empty groups, duplicate rules, and files over Google's 500 KB limit.
Accurate URL testing
The tester follows the robots.txt standard (RFC 9309) and Google's documented rules, so its answers match how real crawlers behave:
- Consecutive
User-agentlines share one set of rules, and a crawler uses the most specific group that names it, falling back to*only if none does. - Googlebot-Image and Googlebot-News follow the Googlebot group when they don't have their own, and Applebot follows Googlebot's rules if no Applebot group exists.
- When several rules match, the longest one wins, and Allow wins a tie.
*matches any characters,$marks the end of the URL, paths are case-sensitive, query strings count, and encoded characters like%C3%A9match their unencoded form.
Frequently asked questions
What's the difference between "Disallow: /" and "Disallow:"?
Disallow: / blocks the entire site for that crawler. Disallow: with nothing after it blocks nothing, so the crawler may fetch everything. They look almost the same and do opposite things.
Why can't it load my robots.txt from my site?
For security, browsers only let a web page read files from another website when that site allows it, and most sites don't allow it for robots.txt. If loading fails, the tool gives you a link to open the file. Copy everything there and paste it into the box.
Does blocking a page in robots.txt remove it from Google?
Not necessarily. Robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. To keep a page out of search results, let it be crawled and add a noindex robots meta tag or X-Robots-Tag header.
How do I block AI crawlers?
Add a group naming the AI crawlers you want to block, like GPTBot, ClaudeBot, Google-Extended, CCBot and PerplexityBot, followed by Disallow: /. Load the Block AI crawlers sample to see an example, then test it with the crawlers in the tester's AI section.
Is my robots.txt sent to Flipper File?
No. Checking and testing happen in your browser. When you load a file from a site, your browser requests it directly from that site.