Robots.txt testerPaste your file, pick a page, and see which search engine and AI bots can read it.
The robots.txt file tells bots what they may crawl on your site. One badly written line, and Google or an AI stops reading your pages. This free tool tests your file for 15 bots, in your browser. Nothing is sent.
Test your robots.txt file
Open your site’s address followed by /robots.txt in your browser, for example yoursite.com/robots.txt. Copy all the text, and paste it here.
The test runs entirely in your browser. I receive neither your file nor the address you test. It applies the rules of the RFC 9309 standard and the way Google reads them: the bot’s own group, otherwise the * group, then the longest rule, and in a tie, the least restrictive one.
One limit to know: each bot reads the file in its own way. This test tells you what the rules ask for. It does not guarantee that a bot obeys them. To find out, you need to look at your server log.
What is the robots.txt file for?
To tell bots what they are allowed to crawl. Not to hide a page from Google.
The robots.txt file is a small text file placed at the root of your site. Before crawling your pages, most bots read it. It tells them, bot by bot, which addresses they may open and which ones they should leave alone.
Its rules are described in a standard, RFC 9309, published in September 2022. Google explains that these rules only apply to the exact site where the file sits: same domain name, same protocol, same port. The robots.txt of www.yoursite.com does not apply to shop.yoursite.com.
Google points out that the file is first of all a way to manage how bots move through your site. It is not a way to keep a page out of Google. A blocked page can even show up in the results, without a description, if other sites link to it. To remove a page from Google, Google recommends the noindex tag or a password.
This is a point I often see misunderstood. People block a page in robots.txt thinking they are hiding it. In reality, they only stop Google from reading it, including the noindex instruction it contains.
Today, robots.txt has a new role: telling AI bots whether they may read your pages. That is why the tool above also tests GPTBot, ClaudeBot, PerplexityBot and the others.
How does the tool decide that a bot is blocked?
The same logic as Google, applied line by line. The result shows the rule that decided.
It looks for the bot’s group
A group starts with one or more User-agent lines. If a group carries the bot’s name, it applies. Otherwise, the * group applies.
It keeps the ones that match
Each Allow or Disallow line is compared with the address being tested. The * sign stands for any characters, and $ marks the end of the address.
The most specific rule wins
If several rules match, the longest one applies. A rule on /blog/ beats a rule on /.
In a tie, Allow wins
If an Allow rule and a Disallow rule have the same length, Google applies the least restrictive one: the page is allowed.
The lines of a robots.txt file, explained
What you need to know to read, or write, your file without mistakes.
The bot’s name
The line User-agent: GPTBot opens a group for GPTBot. The name is not case sensitive.
User-agent: * targets all the bots that have no group under their own name.
What is forbidden
Disallow: /wp-admin/ forbids everything that starts with /wp-admin/.
Disallow: / forbids the whole site. An empty Disallow line forbids nothing.
The exception
Allow: /wp-admin/admin-ajax.php allows one address inside a forbidden folder.
The longest rule wins.
The wildcards
Disallow: /*?sort= forbids all the addresses that contain ?sort=.
Disallow: /*.pdf$ forbids the addresses that end in .pdf.
The site map
Sitemap: followed by the full address of your sitemap. This line applies to all bots.
It does not belong to any group, and you can add several.
The delay between two visits
Google does not support this directive. Bing says it respects it, and Anthropic says its bots follow it.
Do not count on it to slow Googlebot down.
Comments
Everything after a # is ignored. Handy for explaining a rule to the person who will take over your site.
Google also ignores the lines it does not understand.
No mixing with *
For Google, a bot that has its own group follows only that one. The rules of the * group are not added to it.
A common mistake: believing that a GPTBot group inherits the bans of the * group.
The maximum size
Google reads at most 500 KiB of the file. Beyond that, the rest is ignored.
Google also keeps the file in its cache for up to 24 hours in general: a change is not seen right away.
The most common mistakes in a robots.txt
They are easy to make, and easy to fix once you spot them.
The worst one: a forgotten Disallow: /. The whole site is blocked while it is being built, then the line is not removed when the site goes live. No bot reads the site anymore. The test on this page shows it in a second.
Second mistake: a separate group for one bot, thinking it also keeps the rules of the * group. For Google, it does not. If you create a GPTBot group, copy into it the bans that must also apply to it.
Third mistake: blocking files that pages need, such as style sheets or images. Google advises against blocking files without which a page becomes hard to understand.
Last mistake: the wrong site. The file on www.yoursite.com does not apply to yoursite.com without the www, nor to a subdomain. Each site address must have its own, or redirect to the right one.
The WordPress robots.txt
WordPress creates a robots.txt file on its own. Here is what it contains, and how to change it.
If you have not uploaded a robots.txt file to your web host, WordPress builds one on the fly. According to the WordPress documentation, it contains three lines: User-agent: *, a Disallow line for the admin folder, often /wp-admin/, and an Allow line for the admin-ajax.php file in that folder.
Since version 5.3, the “Discourage search engines from indexing this site” checkbox no longer adds Disallow: /. WordPress uses a noindex tag in the pages instead. That is more accurate: Google can read the instruction.
To change your robots.txt, there are two ways. Upload a real robots.txt file to the root of your site, which then replaces the WordPress one. Or use a plugin that lets you edit it from the admin area. Either way, test the result with the tool on this page.
I explain how the whole file works, with examples, in my guide to the robots.txt file (in French). And to see what bots really do on your site, see the page Googlebot in your logs.
Blocking AI bots: which line, what effect?
What each company says about its bot, according to its documentation, checked on October 23, 2026.
| Bot | The line to write | What it changes, according to the company |
|---|---|---|
| GPTBot | User-agent: GPTBot then Disallow: / | OpenAI: your text should not be used for training |
| OAI-SearchBot | User-agent: OAI-SearchBot then Disallow: / | OpenAI: it no longer reads your pages for ChatGPT search |
| ClaudeBot | User-agent: ClaudeBot then Disallow: / | Anthropic: it no longer collects your pages for training |
| PerplexityBot | User-agent: PerplexityBot then Disallow: / | Perplexity: it no longer reads your pages for its search index |
| Google-Extended | User-agent: Google-Extended then Disallow: / | Google: your text is no longer used for Gemini, with no effect on Google Search |
| Applebot-Extended | User-agent: Applebot-Extended then Disallow: / | Apple: your text is no longer used to train its models, and you stay in its results |
Two bots read on a person’s request: ChatGPT-User and Perplexity-User. OpenAI and Perplexity state that the robots.txt file may not apply to them. Anthropic, for its part, says that Claude-User respects it.
On my own site, I do not block any of the major AI bots (OpenAI, Anthropic, Perplexity, Google, Apple, Mistral). I recommend looking at your log first for a few weeks: who reads your pages, which ones, and whether people arrive afterwards. It is all explained on the AI bots page.
Free to start, then €9 to €99 a month, excl. VAT
One free plan, then four paid plans. Prices exclude VAT. Pay yearly and two months are free.
free plan
€0forever
No credit card.
1 site
1 user
30 days of history
subscription
€990excl. VAT per monthexcl. VAT per year
or €90 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
1 site
1 user
3 months of history
7-day free trial
€19190excl. VAT per monthexcl. VAT per year
or €190 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
3 sites
1 user
13 months of history
subscription
€39390excl. VAT per monthexcl. VAT per year
or €390 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
10 sites
3 users
25 months of history
Read-only access for 5 clients
subscription
€99990excl. VAT per monthexcl. VAT per year
or €990 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
50 sites
5 users
25 months of history
Read-only access for 50 clients
Every new account tries the Standard plan for 7 days, no credit card needed. After that, you pick a subscription, or stay on the free Starter plan.
How do I test my robots.txt file?
Open yoursite.com/robots.txt in your browser, copy the text, and paste it into the tool on this page. Enter the address to test, then click “Test”.
You see, for 15 bots, whether the address is allowed or blocked, and which line decided.
Where is the robots.txt file?
Always at the root of the site: yoursite.com/robots.txt. A file placed in a subfolder is not read.
On WordPress, if there is no real file, WordPress creates one on the fly.
Does robots.txt stop a page from showing up in Google?
No. Google says it clearly: it is not a way to keep a page out of Google. A blocked page can show up without a description if other sites link to it.
To remove a page, Google recommends the noindex tag, or a password. And the page must not be blocked in robots.txt, otherwise Google does not read the instruction.
How do I block GPTBot?
Add a group under its name: User-agent: GPTBot, then Disallow: / on the next line.
Then test the file here. Careful: if you already have a * group, GPTBot will only follow its own group.
Why does my test say “allowed” when I blocked the folder?
Check three things. The bot’s name in the User-agent line. The start of the address, which must match exactly, capital letters included. And a possible longer Allow rule, which wins.
The tool’s result shows the line that decided: it is the quickest way to understand.
Do all bots respect robots.txt?
No. It is an instruction, not a lock. The bots of the major search engines and AI companies say they respect it, with exceptions for reads made on a person’s request.
Malicious bots ignore it. To know what is really going on, you need to look at your server log.
How long before Google sees my new robots.txt?
Google says it keeps the file in its cache for up to 24 hours in general. OpenAI also mentions about 24 hours for its search.
Do not panic if a change does not show within the hour.
Should I block /wp-admin/ on WordPress?
WordPress already does it in its default file, with an exception for admin-ajax.php, which some public pages use.
If you write your own file, keep those two lines.
Does this test save my file?
No. Everything happens in your browser. I receive neither your file nor the address you test.
You can close the page: nothing is kept.
Can I test another site’s robots.txt?
Yes. Open its address followed by /robots.txt, copy the text and paste it into the tool. A site’s robots.txt file is public.
It is a good way to see how other sites treat AI bots.
What is the difference between Disallow and noindex?
Disallow, in robots.txt, asks the bot not to read the page. Noindex, in the page, asks the search engine not to show it in its results.
For a search engine to see the noindex instruction, it must be able to read the page. So do not combine the two on the same address.
Let the analyzer watch your robots.txt.
It reads it every day, and warns you in the morning if it starts blocking an AI.
You create your account with your email. It’s free.
You confirm your address. The 7-day Standard trial starts, no credit card needed.
You connect your site. You see what bots really do on it.
Page published on November 4, 2026 · Last updated on November 4, 2026
Start free