Log analyzer · AI bots

AI crawlersWho reads your pages: GPTBot, ClaudeBot, PerplexityBot, and the ones that read live.

AI companies send bots to read the web. Some learn, some fill an index, some read while a person is asking a question. Your log sees them all. My analyzer tells you which ones, and on which pages.

Names verifiedFake bots unmaskedPages read by AI

Why track AI crawlers in your logs?

Because it is the only place where you can see them. Your usual stats do not show them.

AI crawlers do not behave like visitors. They ask for the page, take the text, and leave. They do not run the little scripts that count visits. A study published by Vercel in December 2024 checked this: none of the major AI crawlers it observed executed the JavaScript on pages.

The result: Google Analytics does not see them. Most statistics plugins do not either. The server log, on the other hand, notes every one of their visits, with their name and their address.

On my own site, I did the math in July 2026. AI bots had read my pages 6,516 times, for 42 visitors sent back to me. I told the story in my article on the log analysis of my site (in French). That number made me understand one thing: you need to know who is reading before you ask who is clicking.

Not all AI crawlers do the same job. Some read to train a future model. Others read to fill the index of an answer engine. Others read live, because a person has just asked a question. That last case is the most interesting: it shows that a real person is interested in your topic, right at that moment.

My analyzer separates these three uses. It also checks that each bot is really the one it claims to be. Then it shows you which pages AI reads, and which ones it misses.

Three families of AI crawlers

A bot’s name often tells you what it is for. The companies describe them in their official documentation.

1 · training

They learn

GPTBot at OpenAI, ClaudeBot at Anthropic, CCBot at Common Crawl. They collect text that may be used to train future models.

They visit often, and on many pages. They do not send visitors back.

2 · search index

They fill a search engine

OAI-SearchBot for ChatGPT search, Claude-SearchBot at Anthropic, PerplexityBot at Perplexity. They read your pages so they can suggest them later in an answer.

It is the equivalent of Googlebot, for AI search engines.

3 · live reads

They read for a person

ChatGPT-User, Claude-User, Perplexity-User. They open a page because a person has just asked the AI a question, or given it a link.

Each visit matches a real request, happening right now.

The main AI crawlers, and what they do

Based on each company’s official documentation, checked on October 23, 2026.

BotCompanyWhat it is forThe robots.txt file
GPTBotOpenAICollect text for trainingRespected
OAI-SearchBotOpenAIPower ChatGPT searchRespected
ChatGPT-UserOpenAIRead a page at a user’s requestMay not apply, according to OpenAI
ClaudeBotAnthropicCollect text for trainingRespected
Claude-SearchBotAnthropicImprove search qualityRespected
Claude-UserAnthropicRead a page at a user’s requestRespected, according to Anthropic
PerplexityBotPerplexityFill the search index, not for trainingRespected
Perplexity-UserPerplexityRead a page at a user’s requestGenerally ignored, according to Perplexity
CCBotCommon CrawlBuild a public archive of the webRespected

OpenAI, Anthropic, Perplexity, Apple and Common Crawl publish the list of their bots’ addresses. My analyzer uses it to unmask impostors, the way it does for Googlebot.

Two names come up often without being bots. Google-Extended and Applebot-Extended are instructions you write in the robots.txt file. They let you refuse to have your text train Google’s or Apple’s models. Google says Google-Extended has no browser name of its own: you will never see it in your log. Apple says Applebot-Extended does not crawl pages.

Meta also has its own bots, Meta-ExternalAgent and Meta-ExternalFetcher. The second one reads a page at a user’s request, and Meta says it may not follow the robots.txt file.

What the tool shows you about AI

An example on a made-up site. The bot names are real, the numbers are invented.

atelier-exemple.fr · Artificial intelligence · 30 daysDemo · made-up numbers
1,150reads to train an AI
530reads for an AI search engine’s index
156live reads, during a conversation
47visits from people who came from an AI

Reads by bot

GPTBotOpenAI, training610
ClaudeBotAnthropic, training540
OAI-SearchBotOpenAI, search320
PerplexityBotPerplexity, index210
ChatGPT-Userlive read95
Fake ClaudeBotaddress not on the official list38

What the tool spots

/stages-ete/ is your most-read page for AI. 88 reads, and then 14 visits from people who came from an AI.

/blog/choisir-son-argile/ is read, but never visited. AI reads it, but nobody arrives through it.

12 pages return an error to AI crawlers. These are old addresses, still linked somewhere.

A demo on a fictional site. These numbers don’t come from any customer.

What you can track, bot by bot

Everything is sorted by use: training, search index, live reads.

01 · the reads

The number of reads, bot by bot

Each AI bot, with its company, its use and its number of visits, day by day.

02 · the pages

The pages AI is interested in

The list of your pages most read by AI. And the ones no AI has ever opened.

03 · the visits

The people who arrive from an AI

Visits from ChatGPT, Perplexity, Claude or Gemini. OpenAI says ChatGPT adds a tag to its links: the tool recognizes it.

04 · the journey

Read, then visited

For each page: was it read by an AI, then visited by a person who came from an AI? You see the pages that lead people to you.

05 · the impostors

Fake AI bots

A bot that calls itself ClaudeBot or GPTBot, but whose address is not on the official list, is set aside.

06 · the access rules

What your robots.txt allows

The tool reads your robots.txt file and tells you which AI bots it blocks. It warns you in the morning if that changes, or if a new AI shows up.

Should you block AI crawlers?

It is a real question, and the answer depends on you. Here is how I frame it.

Blocking a bot means writing it in your robots.txt file. For example, User-agent: GPTBot followed by Disallow: / asks GPTBot not to read any page. OpenAI says that blocking GPTBot signals that your text should not be used for training.

But not all bots are equal. Blocking a training bot does not make you disappear from answers. Blocking a search bot, such as OAI-SearchBot, can on the other hand remove you from that engine’s results. OpenAI says a change to the robots.txt file takes about 24 hours to apply to its search.

And some bots read anyway. OpenAI writes that the robots.txt rules may not apply to ChatGPT-User, because the read is requested by a person. Perplexity writes that Perplexity-User generally ignores them. Anthropic, for its part, says its three bots respect them.

On my own site, I do not block any of the major AI bots (OpenAI, Anthropic, Perplexity, Google, Apple, Mistral). Before you decide anything, I recommend watching your log for a few weeks. Who reads your pages? Which ones? Do people arrive afterwards? Then you can decide bot by bot, knowing the numbers.

If you want to review your robots.txt file, I explain how it works in my guide to the robots.txt file (in French). And the analyzer tells you, page by page, what your file allows. For a test right now, paste your file into the free robots.txt test.

What the log says about AI, and what it does not

I prefer to be clear: a read is not a citation.

What the log shows

Which AI bots read your pages, when, and how many times.

The use behind each read: training, index, or live read.

The people who arrive on your site from an AI, when the AI passes that information along.

The errors your pages return to AI bots.

What it does not show

Whether an AI cites you in its answer. The log sees the read, not what the AI does with it.

How many times a question is asked to an AI. Nobody publishes that number.

Every visit that came from an AI. Some arrive with no indication at all, and look like direct visits.

Pricing

Free to start, then €9 to €99 a month, excl. VAT

One free plan, then four paid plans. Prices exclude VAT. Pay yearly and two months are free.

Starter

free plan

€0forever

No credit card.

1 site

1 user

30 days of history

Lite

subscription

€990excl. VAT per monthexcl. VAT per year

or €90 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

1 site

1 user

3 months of history

StandardRecommended

7-day free trial

€19190excl. VAT per monthexcl. VAT per year

or €190 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

3 sites

1 user

13 months of history

Advanced

subscription

€39390excl. VAT per monthexcl. VAT per year

or €390 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

10 sites

3 users

25 months of history

Read-only access for 5 clients

Enterprise

subscription

€99990excl. VAT per monthexcl. VAT per year

or €990 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

50 sites

5 users

25 months of history

Read-only access for 50 clients

Every new account tries the Standard plan for 7 days, no credit card needed. After that, you pick a subscription, or stay on the free Starter plan.

Questions about AI crawlers

A question about a specific bot? Write to me.

Email me

What is GPTBot?

It is OpenAI’s bot that collects text from the web to train its models. It shows up as GPTBot in your log.

OpenAI publishes the list of its addresses. My analyzer uses it to check that the GPTBot visiting your site is the real one.

How do I know if ChatGPT reads my site?

Look at your server log. OpenAI’s reads show up there under three names: GPTBot, OAI-SearchBot and ChatGPT-User.

My analyzer sorts them for you, checks their addresses and shows you the pages read, day by day.

What is the difference between GPTBot and ChatGPT-User?

GPTBot collects text for training. It comes on its own, on many pages.

ChatGPT-User opens a page because a ChatGPT user asks for it, for example by pasting your link. So each visit matches a real person.

How do I block GPTBot on my site?

Add two lines to your robots.txt file: User-agent: GPTBot, then Disallow: /. OpenAI says this setting signals that your text should not be used for training.

Remember to check your log afterwards that the visits stop. My analyzer also shows you what your file allows.

Do AI crawlers respect the robots.txt file?

The training and search bots from OpenAI, Anthropic and Perplexity say they respect it. Live reads are different: OpenAI and Perplexity say their on-demand bots may not follow it.

The log lets you check for yourself what gets through on your site, bot by bot.

Why does Google Analytics not show AI crawlers?

Because it counts with a script that runs in the browser. AI crawlers do not run it.

Only the server log notes their visits. That is why I built the analyzer.

How do I recognize a visit from ChatGPT?

OpenAI says ChatGPT adds the tag utm_source=chatgpt.com to the links it suggests in its search answers. The analyzer recognizes it.

But not every visit from an AI carries a tag. Some arrive as direct visits. The “read, then visited” journey helps you spot them.

Can someone pretend to be ClaudeBot or GPTBot?

Yes. Any program can write those names. That is why the companies publish the list of their addresses.

The analyzer compares each bot with those lists, updated every night. Impostors are set apart, with the fake bots.

If an AI reads my page, will it cite me?

Not necessarily. A read is a condition, not a guarantee. The AI can read your page and answer without citing it.

The log shows you the reads. Visits from an AI show you the times a person clicked through to you.

Can AI crawlers slow down my site?

They sometimes visit very often, especially the training bots. On a small hosting plan, that can weigh. The log shows you which ones visit the most, and at what time.

Anthropic says its bots respect the Crawl-delay instruction in the robots.txt file, which asks bots to space out their visits. If a bot really bothers you, you can also block it.

What does an error served to an AI bot mean?

The bot asked for a page that no longer exists, or that moved without a redirect. Often, it is an old address still linked somewhere.

The tool lists these errors. A redirect to the right page is usually enough to fix them.

See which AI reads your site.

The beta is open to everyone. Your log has already noted everything.

1

You create your account with your email. It’s free.

2

You confirm your address. The 7-day Standard trial starts, no credit card needed.

3

You connect your site. AI reads show up within the hour.

Page published on November 4, 2026 · Last updated on November 4, 2026

Start free