Google: Most Sites Don’t Need to Worry About Crawl Budget

Google says a ‘substantial’ segment of site owners should care about crawl budget, but the ‘vast majority’ do not need to.

Google’s Search Relations team says most sites don’t need to worry about crawl budget in the latest episode of the Search Off the Record podcast.

Gary Illyes of Google discussed the topic in length, saying the team has been pushing back on their previous messaging, and adding that a ‘substantial segment’ does have to care about it.

However, crawl budget shouldn’t be a concern for a majority of sites, Illyes explains:

“We’ve been pushing back on the crawl budget, historically, typically telling people that you don’t have to care about it.

And I stand my ground and I still say that most people don’t have to care about it. We do think that there is a substantial segment of the ecosystem that has to care about it.

…but I still believe that – I’m trying to reinforce this here – that the vast majority of the people don’t have to care about it.”

In an effort to clarify previous messaging, Google has recently been publishing more information about crawl budget.

For example, just last month Google dedicated a whole episode of its SEO Mythbusting YouTube series to the topic of crawl budget.

When to Care About Crawl Budget / When Not To Care

SEOs typically want to hear a hard number when it comes to crawl budget – such as your site has to have X-number of pages before crawl budget is a concern.

If forced to give a number, Illyes says roughly a million URLs is the baseline before a site owner really needs to care about crawl budget.

Sites with fewer than a million URLs do not have to care about crawl budget.

Factors Affecting Crawl Budget

For sites with over a million URLs, these are some of the factors that could lead to, or indicate, crawl budget issues.

Factor 1: Pages not crawled in a long time

“What would I look at? Probably URLs that were never crawled. That’s a good indicator for how well discovered, how well crawled a site is…

So I would look at pages that were never crawled. For this you probably want to look at your server logs because that can give you the absolute truth.”

Factor 2: Widespread changes after long periods of time

“Then I would also look at the refresh rates. Like if you see that certain parts of the site were not refreshed for a long period of time, say months, and you did make changes to pages in that section, then you probably want to start thinking about crawl budget.”

Fixing Crawl Budget Issues

Illyes offers two suggesting for fixing crawl budget issues.

First, try removing non-essential pages. Every page Googlebot has to crawl reduces the crawl budget for other pages.

So an excessive amount of “gibberish” content could lead to important content not getting crawled.

Illyes’ second suggestion is to avoid sending “back off” signals to Googlebot.

Back off signals are certain sever codes that will tell Googlebot to immediately stop crawling a site.

Source: https://www.searchenginejournal.com/google-most-sites-dont-need-to-worry-about-crawl-budget/378729/