Log analyzer · crawl budget

Crawl budgetWhat Google really crawls on your site, and why it is rarely a problem for a small site.

You read everywhere that you should “optimize your crawl budget”. Google itself writes that its guide is mainly for very large sites. Your server log shows what Googlebot really does on your site. And it tells you whether you have a problem or not.

Verified visitsWaste spottedPlain verdict

What is crawl budget?

It is the number of pages Google can and wants to crawl on your site. Two things set it.

Googlebot does not read the whole web at once. It visits each site within a limit. Google calls that limit the crawl budget.

Google defines it as the set of URLs it can crawl, and wants to crawl. So there are two limits. The first is your server’s capacity. Google does not want to crash it. It adjusts how many connections it opens at the same time, and for how long. This limit is shared by all its bots.

The second is interest. Google calls it crawl demand. If your site changes little, or if its pages are of little interest, Google crawls less. It says so itself: when demand is low, it crawls your site less.

So your budget is the smaller of the two. A fast server is not enough if Google is not interested. And interest is not enough if the server struggles.

Among SEO specialists, people talk about it a lot. Yet for most sites I come across, it is not the real problem. The rest of this page tells you why, and how to check in your log.

Who is crawl budget for?

Google answers this question itself, at the start of its guide.

Google’s crawl budget guide targets three cases. Very large sites, with over one million pages, whose content changes every week. Medium or large sites, with over 10,000 pages, whose content changes every day. And sites with many URLs classed as “Discovered, currently not indexed” in Search Console.

Google says these figures are rough orders of magnitude, not exact thresholds. But the idea is clear. A 20-page brochure site, a blog of 300 posts or a small shop is not the target.

For these sites, Google mainly advises keeping a sitemap up to date and following the indexing report in Search Console. That is all.

Let me be frank: if your site has a few hundred pages, I advise against spending time on “optimizing your crawl budget”. Spend that time fixing the errors Googlebot runs into, and linking your pages to each other. That is where the real gains are.

There is one exception. A small shop with filters can create thousands of URLs without meaning to: one per color, per size, per sort order. For Google, each one is a page. That is where the log becomes very useful.

Does your site have a crawl budget problem?

A few common situations, read in the light of Google’s guide.

Your siteShould you worry?What I look at in the log
A brochure site of 10 to 50 pagesNo, usuallyThe errors served to Googlebot, and the pages it never opens
A blog of a few hundred postsRarelyOld URLs that still answer 404, and chained redirects
A shop with filters and sort optionsSometimesThe number of URLs with parameters that Googlebot opens each day
A site of over 10,000 pages that changes every dayYesThe share of visits that land on useful pages, day after day

Google’s guide was last updated on July 22, 2026. It mentions “Discovered, currently not indexed”: in Search Console, Google explains that the crawl may have been postponed because the site was at risk of being overloaded.

What wastes Googlebot’s crawling

Even a small site can waste Googlebot’s time. Here is what Google lists, and what your log shows about it.

01 · fake pages

Disguised errors

A page that says “not found” but answers as if all is well. Google calls this a soft 404. It writes that these pages keep being crawled, and waste the budget.

02 · chains

Redirects in a row

One URL redirects to another, which redirects again. Google advises avoiding long chains, which get in the way of crawling. The log shows them step by step.

03 · duplicates

The same content at several URLs

With and without a trailing slash, with a sort parameter, as a printable version. Google advises eliminating duplicate content.

04 · filters

URLs with parameters

A shop’s filters create one URL per combination. Google explains elsewhere that faceted navigation often creates duplicate content, and can lead to a site being over-crawled.

05 · removed pages

Pages that no longer exist

For a page removed for good, Google advises answering 404 or 410. It writes that a 404 is a strong signal to stop crawling that URL.

06 · slowness

A server that struggles

If your server answers slowly or goes down, Google lowers its capacity limit. The log shows the failures served to Googlebot, and the tool warns you.

What the tool shows you about crawling

An example on a fictional site. The numbers are made up.

atelier-exemple.fr · Search engine bots · 30 daysDemo · made-up figures
3,000visits from the real Googlebot
78%on pages that answer normally
14%on redirects
8%on errors

URLs that waste Googlebot’s time

/boutique/?tri=prixsort parameter142
/tarifs-2024/answers 40486
/ancien-blog/301 redirect63
/cours-enfants/answers 40438

What the tool spots

/tarifs-2024/ answers 404 to Googlebot. 86 visits this month, all wasted. A redirect to the pricing page would fix it.

3,000 visits, against 2,640 the month before. Googlebot is visiting a little more often since your latest updates.

140 fake Googlebots unmasked. If they were counted, they would have suggested much heavier crawling.

A demo on a fictional site. These numbers don’t come from any customer.

Look at the last alert. Many bots pretend to be Googlebot. If you count them, you believe Google is crawling your site heavily. That is wrong. My analyzer checks every visit against Google’s official lists. I explain it on the page Googlebot in your logs.

Search Console or server log: how do you measure crawling?

Both are useful. They do not show the same thing.

Search Console has a report called “Crawl stats”. It shows the number of requests from Google, file sizes, response time, and a breakdown by type of response. Google says the example URLs it gives are not complete. It also says sites with fewer than 1,000 pages generally do not need this report.

Your log, on the other hand, records every visit, without exception. You see which page Googlebot opens, at what time, and what your server answers. You also see what Search Console never shows: fake Googlebots.

My analyzer does the sorting. It keeps only the visits of the real Googlebot. It counts the useful pages, the redirects, the errors. And it compares with previous weeks, to see whether crawling is going up or down.

If you want to understand Search Console in detail, I cover it in my analytics training (in French). To read a log line yourself, see the page on how the analyzer works.

What I recommend for a small site

Four simple steps. They help Googlebot, and your visitors too.

01 · The sitemap

Keep your sitemap up to date

This is Google’s first piece of advice for sites that are not huge. On WordPress, a plugin like Rank Math does it on its own.

02 · The errors

Fix what Googlebot runs into

An old URL giving a 404, a page returning a server error. Start with the ones Googlebot opens most often. See the page on 404 errors.

03 · The redirects

Redirect in a single hop

An old URL should lead straight to the right page. If it goes through two or three steps, fix the redirect, and the links that point to it.

04 · The links

Link your important pages

Google mostly finds pages through the links on other pages. An important page must receive links. See the page on internal linking.

Pricing

Free to start, then €9 to €99 a month, excl. VAT

One free plan, then four paid plans. Prices exclude VAT. Pay yearly and two months are free.

Starter

free plan

€0forever

No credit card.

1 site

1 user

30 days of history

Lite

subscription

€990excl. VAT per monthexcl. VAT per year

or €90 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

1 site

1 user

3 months of history

StandardRecommended

7-day free trial

€19190excl. VAT per monthexcl. VAT per year

or €190 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

3 sites

1 user

13 months of history

Advanced

subscription

€39390excl. VAT per monthexcl. VAT per year

or €390 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

10 sites

3 users

25 months of history

Read-only access for 5 clients

Enterprise

subscription

€99990excl. VAT per monthexcl. VAT per year

or €990 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free

50 sites

5 users

25 months of history

Read-only access for 50 clients

Every new account tries the Standard plan for 7 days, no credit card needed. After that, you pick a subscription, or stay on the free Starter plan.

Crawl budget questions

A question about how your site is crawled? Write to me.

Email me

What is crawl budget?

It is the number of URLs Google can and wants to crawl on your site. Google sets it according to your server’s capacity and its interest in your pages.

In French it is also called budget crawl, budget de crawl or budget d’exploration. All four names mean the same thing.

How do I find my site’s crawl budget?

Google does not give a figure. You can only measure actual crawling: how many pages Googlebot opens, and how often.

To do that, look at your server log, or the “Crawl stats” report in Search Console. My analyzer does the calculation every day, keeping only the real Googlebot.

Does my small site need to optimize its crawl budget?

Usually not. Google’s guide is aimed at very large sites, and at sites of over 10,000 pages that change every day.

For a small site, keep a sitemap up to date, fix the errors and link your pages. That is enough.

How do I increase crawl budget?

You cannot raise it on demand. You avoid wasting it: fewer errors, no redirect chains, no duplicates.

A fast, stable server helps too: Google reduces its crawling when your server slows down or goes down.

Do 404 errors waste crawl budget?

A real 404, on a deleted page, is even a good signal. Google writes that a 404 strongly tells it to stop crawling that URL.

What wastes it is fake pages: the ones that say “not found” but answer like a normal page. Google calls them soft 404s.

What does “Discovered, currently not indexed” mean?

It is a status in the Search Console indexing report. Google knows the URL, but has not crawled it yet.

Google explains that the crawl may have been postponed because your site was at risk of being overloaded. If many of your pages are in this state, the crawl budget guide applies to you.

Do fake Googlebots distort the numbers?

Yes, if you do not sort them out. Many bots pretend to be Googlebot. Counted by mistake, they suggest much heavier crawling than there really is.

My analyzer checks every visit against the IP address lists published by Google, and sets the impostors aside.

Does crawl budget apply to AI bots?

Google’s guide is about its own bots. AI bots have their own rules, and their companies do not publish a budget.

Your log still shows their visits, page by page. That is the subject of the page on AI bots.

Is crawl budget related to my site’s speed?

Yes, through the capacity limit. Google adjusts its crawling to the health of your server. Failures or slow responses push it to slow down.

The log shows the moments when your server sent errors to Googlebot. The tool’s technical analysis also measures the speed of your pages.

How often does Googlebot visit a small site?

There is no normal figure. It depends on the number of pages, how often you change them and how interested Google is in your site.

The most useful thing is to follow the trend. The analyzer compares each month with the previous one, and warns you if things change a lot.

See what Googlebot does on your site.

No theory: your real visits, sorted, with a clear verdict.

1

You create your account with your email. It’s free.

2

You confirm your address. The 7-day Standard trial starts, no credit card needed.

3

You connect your site. Googlebot’s crawling shows up within the hour.

Page published on November 4, 2026 · Last updated on November 4, 2026

Start free