Crawl budgetWhat Google really crawls on your site, and why it is rarely a problem for a small site.
You read everywhere that you should “optimize your crawl budget”. Google itself writes that its guide is mainly for very large sites. Your server log shows what Googlebot really does on your site. And it tells you whether you have a problem or not.
What is crawl budget?
It is the number of pages Google can and wants to crawl on your site. Two things set it.
Googlebot does not read the whole web at once. It visits each site within a limit. Google calls that limit the crawl budget.
Google defines it as the set of URLs it can crawl, and wants to crawl. So there are two limits. The first is your server’s capacity. Google does not want to crash it. It adjusts how many connections it opens at the same time, and for how long. This limit is shared by all its bots.
The second is interest. Google calls it crawl demand. If your site changes little, or if its pages are of little interest, Google crawls less. It says so itself: when demand is low, it crawls your site less.
So your budget is the smaller of the two. A fast server is not enough if Google is not interested. And interest is not enough if the server struggles.
Among SEO specialists, people talk about it a lot. Yet for most sites I come across, it is not the real problem. The rest of this page tells you why, and how to check in your log.
Who is crawl budget for?
Google answers this question itself, at the start of its guide.
Google’s crawl budget guide targets three cases. Very large sites, with over one million pages, whose content changes every week. Medium or large sites, with over 10,000 pages, whose content changes every day. And sites with many URLs classed as “Discovered, currently not indexed” in Search Console.
Google says these figures are rough orders of magnitude, not exact thresholds. But the idea is clear. A 20-page brochure site, a blog of 300 posts or a small shop is not the target.
For these sites, Google mainly advises keeping a sitemap up to date and following the indexing report in Search Console. That is all.
Let me be frank: if your site has a few hundred pages, I advise against spending time on “optimizing your crawl budget”. Spend that time fixing the errors Googlebot runs into, and linking your pages to each other. That is where the real gains are.
There is one exception. A small shop with filters can create thousands of URLs without meaning to: one per color, per size, per sort order. For Google, each one is a page. That is where the log becomes very useful.
Does your site have a crawl budget problem?
A few common situations, read in the light of Google’s guide.
| Your site | Should you worry? | What I look at in the log |
|---|---|---|
| A brochure site of 10 to 50 pages | No, usually | The errors served to Googlebot, and the pages it never opens |
| A blog of a few hundred posts | Rarely | Old URLs that still answer 404, and chained redirects |
| A shop with filters and sort options | Sometimes | The number of URLs with parameters that Googlebot opens each day |
| A site of over 10,000 pages that changes every day | Yes | The share of visits that land on useful pages, day after day |
Google’s guide was last updated on July 22, 2026. It mentions “Discovered, currently not indexed”: in Search Console, Google explains that the crawl may have been postponed because the site was at risk of being overloaded.
What wastes Googlebot’s crawling
Even a small site can waste Googlebot’s time. Here is what Google lists, and what your log shows about it.
Disguised errors
A page that says “not found” but answers as if all is well. Google calls this a soft 404. It writes that these pages keep being crawled, and waste the budget.
Redirects in a row
One URL redirects to another, which redirects again. Google advises avoiding long chains, which get in the way of crawling. The log shows them step by step.
The same content at several URLs
With and without a trailing slash, with a sort parameter, as a printable version. Google advises eliminating duplicate content.
URLs with parameters
A shop’s filters create one URL per combination. Google explains elsewhere that faceted navigation often creates duplicate content, and can lead to a site being over-crawled.
Pages that no longer exist
For a page removed for good, Google advises answering 404 or 410. It writes that a 404 is a strong signal to stop crawling that URL.
A server that struggles
If your server answers slowly or goes down, Google lowers its capacity limit. The log shows the failures served to Googlebot, and the tool warns you.
What the tool shows you about crawling
An example on a fictional site. The numbers are made up.
URLs that waste Googlebot’s time
What the tool spots
/tarifs-2024/ answers 404 to Googlebot. 86 visits this month, all wasted. A redirect to the pricing page would fix it.
3,000 visits, against 2,640 the month before. Googlebot is visiting a little more often since your latest updates.
140 fake Googlebots unmasked. If they were counted, they would have suggested much heavier crawling.
A demo on a fictional site. These numbers don’t come from any customer.
Look at the last alert. Many bots pretend to be Googlebot. If you count them, you believe Google is crawling your site heavily. That is wrong. My analyzer checks every visit against Google’s official lists. I explain it on the page Googlebot in your logs.
Search Console or server log: how do you measure crawling?
Both are useful. They do not show the same thing.
Search Console has a report called “Crawl stats”. It shows the number of requests from Google, file sizes, response time, and a breakdown by type of response. Google says the example URLs it gives are not complete. It also says sites with fewer than 1,000 pages generally do not need this report.
Your log, on the other hand, records every visit, without exception. You see which page Googlebot opens, at what time, and what your server answers. You also see what Search Console never shows: fake Googlebots.
My analyzer does the sorting. It keeps only the visits of the real Googlebot. It counts the useful pages, the redirects, the errors. And it compares with previous weeks, to see whether crawling is going up or down.
If you want to understand Search Console in detail, I cover it in my analytics training (in French). To read a log line yourself, see the page on how the analyzer works.
What I recommend for a small site
Four simple steps. They help Googlebot, and your visitors too.
Keep your sitemap up to date
This is Google’s first piece of advice for sites that are not huge. On WordPress, a plugin like Rank Math does it on its own.
Fix what Googlebot runs into
An old URL giving a 404, a page returning a server error. Start with the ones Googlebot opens most often. See the page on 404 errors.
Redirect in a single hop
An old URL should lead straight to the right page. If it goes through two or three steps, fix the redirect, and the links that point to it.
Link your important pages
Google mostly finds pages through the links on other pages. An important page must receive links. See the page on internal linking.
Free to start, then €9 to €99 a month, excl. VAT
One free plan, then four paid plans. Prices exclude VAT. Pay yearly and two months are free.
free plan
€0forever
No credit card.
1 site
1 user
30 days of history
subscription
€990excl. VAT per monthexcl. VAT per year
or €90 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
1 site
1 user
3 months of history
7-day free trial
€19190excl. VAT per monthexcl. VAT per year
or €190 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
3 sites
1 user
13 months of history
subscription
€39390excl. VAT per monthexcl. VAT per year
or €390 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
10 sites
3 users
25 months of history
Read-only access for 5 clients
subscription
€99990excl. VAT per monthexcl. VAT per year
or €990 a year, excl. VAT, 2 months free10 months paid out of 12, 2 months free
50 sites
5 users
25 months of history
Read-only access for 50 clients
Every new account tries the Standard plan for 7 days, no credit card needed. After that, you pick a subscription, or stay on the free Starter plan.
What is crawl budget?
It is the number of URLs Google can and wants to crawl on your site. Google sets it according to your server’s capacity and its interest in your pages.
In French it is also called budget crawl, budget de crawl or budget d’exploration. All four names mean the same thing.
How do I find my site’s crawl budget?
Google does not give a figure. You can only measure actual crawling: how many pages Googlebot opens, and how often.
To do that, look at your server log, or the “Crawl stats” report in Search Console. My analyzer does the calculation every day, keeping only the real Googlebot.
Does my small site need to optimize its crawl budget?
Usually not. Google’s guide is aimed at very large sites, and at sites of over 10,000 pages that change every day.
For a small site, keep a sitemap up to date, fix the errors and link your pages. That is enough.
How do I increase crawl budget?
You cannot raise it on demand. You avoid wasting it: fewer errors, no redirect chains, no duplicates.
A fast, stable server helps too: Google reduces its crawling when your server slows down or goes down.
Do 404 errors waste crawl budget?
A real 404, on a deleted page, is even a good signal. Google writes that a 404 strongly tells it to stop crawling that URL.
What wastes it is fake pages: the ones that say “not found” but answer like a normal page. Google calls them soft 404s.
What does “Discovered, currently not indexed” mean?
It is a status in the Search Console indexing report. Google knows the URL, but has not crawled it yet.
Google explains that the crawl may have been postponed because your site was at risk of being overloaded. If many of your pages are in this state, the crawl budget guide applies to you.
Do fake Googlebots distort the numbers?
Yes, if you do not sort them out. Many bots pretend to be Googlebot. Counted by mistake, they suggest much heavier crawling than there really is.
My analyzer checks every visit against the IP address lists published by Google, and sets the impostors aside.
Does crawl budget apply to AI bots?
Google’s guide is about its own bots. AI bots have their own rules, and their companies do not publish a budget.
Your log still shows their visits, page by page. That is the subject of the page on AI bots.
Is crawl budget related to my site’s speed?
Yes, through the capacity limit. Google adjusts its crawling to the health of your server. Failures or slow responses push it to slow down.
The log shows the moments when your server sent errors to Googlebot. The tool’s technical analysis also measures the speed of your pages.
How often does Googlebot visit a small site?
There is no normal figure. It depends on the number of pages, how often you change them and how interested Google is in your site.
The most useful thing is to follow the trend. The analyzer compares each month with the previous one, and warns you if things change a lot.
See what Googlebot does on your site.
No theory: your real visits, sorted, with a clear verdict.
You create your account with your email. It’s free.
You confirm your address. The 7-day Standard trial starts, no credit card needed.
You connect your site. Googlebot’s crawling shows up within the hour.
Page published on November 4, 2026 · Last updated on November 4, 2026
Start free