AI crawlersWho reads your pages: GPTBot, ClaudeBot, PerplexityBot, and the ones that read live.
AI companies send bots to read the web. Some learn, some fill an index, some read while a person is asking a question. Your log sees them all. My analyzer tells you which ones, and on which pages.
Why track AI crawlers in your logs?
Because it is the only place where you can see them. Your usual stats do not show them.
AI crawlers do not behave like visitors. They ask for the page, take the text, and leave. They do not run the little scripts that count visits. A study published by Vercel in December 2024 checked this: none of the major AI crawlers it observed executed the JavaScript on pages.
The result: Google Analytics does not see them. Most statistics plugins do not either. The server log, on the other hand, notes every one of their visits, with their name and their address.
On my own site, I did the math in July 2026. AI bots had read my pages 6,516 times, for 42 visitors sent back to me. I told the story in my article on the log analysis of my site (in French). That number made me understand one thing: you need to know who is reading before you ask who is clicking.
Not all AI crawlers do the same job. Some read to train a future model. Others read to fill the index of an answer engine. Others read live, because a person has just asked a question. That last case is the most interesting: it shows that a real person is interested in your topic, right at that moment.
My analyzer separates these three uses. It also checks that each bot is really the one it claims to be. Then it shows you which pages AI reads, and which ones it misses.
Three families of AI crawlers
A bot’s name often tells you what it is for. The companies describe them in their official documentation.
They learn
GPTBot at OpenAI, ClaudeBot at Anthropic, CCBot at Common Crawl. They collect text that may be used to train future models.
They visit often, and on many pages. They do not send visitors back.
They fill a search engine
OAI-SearchBot for ChatGPT search, Claude-SearchBot at Anthropic, PerplexityBot at Perplexity. They read your pages so they can suggest them later in an answer.
It is the equivalent of Googlebot, for AI search engines.
They read for a person
ChatGPT-User, Claude-User, Perplexity-User. They open a page because a person has just asked the AI a question, or given it a link.
Each visit matches a real request, happening right now.
The main AI crawlers, and what they do
Based on each company’s official documentation, checked on October 23, 2026.
| Bot | Company | What it is for | The robots.txt file |
|---|---|---|---|
| GPTBot | OpenAI | Collect text for training | Respected |
| OAI-SearchBot | OpenAI | Power ChatGPT search | Respected |
| ChatGPT-User | OpenAI | Read a page at a user’s request | May not apply, according to OpenAI |
| ClaudeBot | Anthropic | Collect text for training | Respected |
| Claude-SearchBot | Anthropic | Improve search quality | Respected |
| Claude-User | Anthropic | Read a page at a user’s request | Respected, according to Anthropic |
| PerplexityBot | Perplexity | Fill the search index, not for training | Respected |
| Perplexity-User | Perplexity | Read a page at a user’s request | Generally ignored, according to Perplexity |
| CCBot | Common Crawl | Build a public archive of the web | Respected |
OpenAI, Anthropic, Perplexity, Apple and Common Crawl publish the list of their bots’ addresses. My analyzer uses it to unmask impostors, the way it does for Googlebot.
Two names come up often without being bots. Google-Extended and Applebot-Extended are instructions you write in the robots.txt file. They let you refuse to have your text train Google’s or Apple’s models. Google says Google-Extended has no browser name of its own: you will never see it in your log. Apple says Applebot-Extended does not crawl pages.
Meta also has its own bots, Meta-ExternalAgent and Meta-ExternalFetcher. The second one reads a page at a user’s request, and Meta says it may not follow the robots.txt file.
What the tool shows you about AI
An example on a made-up site. The bot names are real, the numbers are invented.
Reads by bot
What the tool spots
/stages-ete/ is your most-read page for AI. 88 reads, and then 14 visits from people who came from an AI.
/blog/choisir-son-argile/ is read, but never visited. AI reads it, but nobody arrives through it.
12 pages return an error to AI crawlers. These are old addresses, still linked somewhere.
A demo on a fictional site. These numbers don’t come from any customer.
What you can track, bot by bot
Everything is sorted by use: training, search index, live reads.
The number of reads, bot by bot
Each AI bot, with its company, its use and its number of visits, day by day.
The pages AI is interested in
The list of your pages most read by AI. And the ones no AI has ever opened.
The people who arrive from an AI
Visits from ChatGPT, Perplexity, Claude or Gemini. OpenAI says ChatGPT adds a tag to its links: the tool recognizes it.
Read, then visited
For each page: was it read by an AI, then visited by a person who came from an AI? You see the pages that lead people to you.
Fake AI bots
A bot that calls itself ClaudeBot or GPTBot, but whose address is not on the official list, is set aside.
What your robots.txt allows
The tool reads your robots.txt file and tells you which AI bots it blocks. It warns you in the morning if that changes, or if a new AI shows up.
Should you block AI crawlers?
It is a real question, and the answer depends on you. Here is how I frame it.
Blocking a bot means writing it in your robots.txt file. For example, User-agent: GPTBot followed by Disallow: / asks GPTBot not to read any page. OpenAI says that blocking GPTBot signals that your text should not be used for training.
But not all bots are equal. Blocking a training bot does not make you disappear from answers. Blocking a search bot, such as OAI-SearchBot, can on the other hand remove you from that engine’s results. OpenAI says a change to the robots.txt file takes about 24 hours to apply to its search.
And some bots read anyway. OpenAI writes that the robots.txt rules may not apply to ChatGPT-User, because the read is requested by a person. Perplexity writes that Perplexity-User generally ignores them. Anthropic, for its part, says its three bots respect them.
On my own site, I do not block any of the major AI bots (OpenAI, Anthropic, Perplexity, Google, Apple, Mistral). Before you decide anything, I recommend watching your log for a few weeks. Who reads your pages? Which ones? Do people arrive afterwards? Then you can decide bot by bot, knowing the numbers.
If you want to review your robots.txt file, I explain how it works in my guide to the robots.txt file (in French). And the analyzer tells you, page by page, what your file allows. For a test right now, paste your file into the free robots.txt test.
What the log says about AI, and what it does not
I prefer to be clear: a read is not a citation.
Which AI bots read your pages, when, and how many times.
The use behind each read: training, index, or live read.
The people who arrive on your site from an AI, when the AI passes that information along.
The errors your pages return to AI bots.
Whether an AI cites you in its answer. The log sees the read, not what the AI does with it.
How many times a question is asked to an AI. Nobody publishes that number.
Every visit that came from an AI. Some arrive with no indication at all, and look like direct visits.
Free to start, then €9 to €99 a month, excl. VAT
One free plan, then four paid plans. Prices exclude VAT. Pay yearly and two months are free.
free plan
€0forever
No credit card.
1 site
1 user
30 days of history
subscription
€990excl. VAT per monthexcl. VAT per year
or €90 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
1 site
1 user
3 months of history
7-day free trial
€19190excl. VAT per monthexcl. VAT per year
or €190 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
3 sites
1 user
13 months of history
subscription
€39390excl. VAT per monthexcl. VAT per year
or €390 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
10 sites
3 users
25 months of history
Read-only access for 5 clients
subscription
€99990excl. VAT per monthexcl. VAT per year
or €990 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
50 sites
5 users
25 months of history
Read-only access for 50 clients
Every new account tries the Standard plan for 7 days, no credit card needed. After that, you pick a subscription, or stay on the free Starter plan.
What is GPTBot?
It is OpenAI’s bot that collects text from the web to train its models. It shows up as GPTBot in your log.
OpenAI publishes the list of its addresses. My analyzer uses it to check that the GPTBot visiting your site is the real one.
How do I know if ChatGPT reads my site?
Look at your server log. OpenAI’s reads show up there under three names: GPTBot, OAI-SearchBot and ChatGPT-User.
My analyzer sorts them for you, checks their addresses and shows you the pages read, day by day.
What is the difference between GPTBot and ChatGPT-User?
GPTBot collects text for training. It comes on its own, on many pages.
ChatGPT-User opens a page because a ChatGPT user asks for it, for example by pasting your link. So each visit matches a real person.
How do I block GPTBot on my site?
Add two lines to your robots.txt file: User-agent: GPTBot, then Disallow: /. OpenAI says this setting signals that your text should not be used for training.
Remember to check your log afterwards that the visits stop. My analyzer also shows you what your file allows.
Do AI crawlers respect the robots.txt file?
The training and search bots from OpenAI, Anthropic and Perplexity say they respect it. Live reads are different: OpenAI and Perplexity say their on-demand bots may not follow it.
The log lets you check for yourself what gets through on your site, bot by bot.
Why does Google Analytics not show AI crawlers?
Because it counts with a script that runs in the browser. AI crawlers do not run it.
Only the server log notes their visits. That is why I built the analyzer.
How do I recognize a visit from ChatGPT?
OpenAI says ChatGPT adds the tag utm_source=chatgpt.com to the links it suggests in its search answers. The analyzer recognizes it.
But not every visit from an AI carries a tag. Some arrive as direct visits. The “read, then visited” journey helps you spot them.
Can someone pretend to be ClaudeBot or GPTBot?
Yes. Any program can write those names. That is why the companies publish the list of their addresses.
The analyzer compares each bot with those lists, updated every night. Impostors are set apart, with the fake bots.
If an AI reads my page, will it cite me?
Not necessarily. A read is a condition, not a guarantee. The AI can read your page and answer without citing it.
The log shows you the reads. Visits from an AI show you the times a person clicked through to you.
Can AI crawlers slow down my site?
They sometimes visit very often, especially the training bots. On a small hosting plan, that can weigh. The log shows you which ones visit the most, and at what time.
Anthropic says its bots respect the Crawl-delay instruction in the robots.txt file, which asks bots to space out their visits. If a bot really bothers you, you can also block it.
What does an error served to an AI bot mean?
The bot asked for a page that no longer exists, or that moved without a redirect. Often, it is an old address still linked somewhere.
The tool lists these errors. A redirect to the right page is usually enough to fix them.
See which AI reads your site.
The beta is open to everyone. Your log has already noted everything.
You create your account with your email. It’s free.
You confirm your address. The 7-day Standard trial starts, no credit card needed.
You connect your site. AI reads show up within the hour.
Page published on November 4, 2026 · Last updated on November 4, 2026
Start free