How Often Should You Perform Technical Website Crawls for SEO?

Any seasoned SEO professional will tell you how important website crawls are for maintaining strong technical SEO. 

But here lies the better questions – how frequently should you perform website crawls? And how often are they actually performing them? 

In this post, we’ll discuss what SEO publications suggest as a “best practice” web crawling cadence and the actual rate SEO pros are performing them. 

Then, I’ll demonstrate the benefits of a ramped-up web crawling cadence by sharing a case study from FOX.com.

What’s a Website Crawl, Anyway? 

Using specialized tools such as Screaming Frog or DeepCrawl, you can take a look “under the hood” of a website – much like a mechanic does when inspecting cars. 

But instead of inspecting the mechanical parts of a car, you are inspecting the optimizable elements of a website – including the quality of its metadata, XML sitemaps, response codes, and more. 

When something isn’t working as expected in SEO, it’s up to you to diagnose the problem and find the solution to fix it.

What Are the Industry Experts Saying About Website Crawl Cadence ‘Best Practice’? 

Industry publications seem to be in agreement that “mini” technical audits should be conducted on a monthly basis and “in-depth” technical audits should be conducted on a quarterly or semi-quarterly basis.

However, there is little chatter specifically discussing the “optimal frequency” for performing website crawls: 

“Part of your ongoing SEO strategy and this Seo Services India includes regular audits to allow you to find and fix issues quickly (we recommend quarterly).” – Erika Varagouli, SEMrush 

“It’s good practice to do an automated scan once a month. This will be often enough to bring up major issues, like any on-page errors in the form of broken links, page titles and meta-data or duplicate content. – Digital Marketing Institute 

“I perform an SEO audit for my clients the first month, monthly (mini-audit), and quarterly (in-depth audit).” – Anna Crowe, Search Engine Journal.

A website crawl and technical audit are not the same thing (thank you for the clear separation of the two, Barry Adams). 

However, it’s fair to assume that these publications would recommend running a website crawl at least as frequently as they run mini-audits on a monthly basis.

What I Learned About SEO Crawl Cadence in ‘Actual Practice’ 

Best practice is one thing. But how often do SEO pros run website crawls for client sites in actual practice? To get an idea, I took to Twitter. (Yes, Twitter polls do have their obvious limitations – but it’s one of the simplest means to get some tangible data.)

In other words, we were all over the map. You may not be too surprised by these poll results. 

After all, both ends of the spectrum could make complete sense – depending on the type and size of the websites you are managing. 

That said, I had an experience at FOX two months ago that made me thankful that we run weekly website crawls across our major domains. I’d like to share it with you all here – in case it encourages you to increase the cadence of your technical website crawls.

Source: https://www.searchenginejournal.com/technical-website-crawls-seo-how-often/380755/

Comments

Post a Comment

Popular posts from this blog

How to Catch & Fix 4 Common SEO Issues Before It’s Too Late

Image
Taking a proactive stance in your SEO processes will help you catch and resolve problems before they impact your rankings. Here's how you can do it. As an SEO professional, it’s your goal – and responsibility – to ensure that you can do what’s necessary to keep things running smoothly and stay up-to-date with your website’s content.  On September 23, I moderated a sponsored Search Engine Journal webinar presented by Steven van Vessum, VP of Community at ContentKing . He shared how to finally take a proactive stance in your SEO processes as you catch (and resolve!) problems before they impact your rankings. Search engines never sleep. They are continuously crawling your site to update their indexes. SEO mistakes can happen anytime.  You need to fix them before they impact your rankings and bottom line. About 80% of SEO issues go unnoticed for at least four weeks .  An average SEO issue can cost up to $75,000 in lost revenue. But with the right tooling and processes, you c...
Read more

Google On Link Algorithms and How Long for Links to Work

Image
In a Webmaster Central hangout , Google’s John Mueller answered a question about when link algorithms are updated and also revealed how long it takes for links to take effect in Google. The original question was concerned with when links take effect and if it was tied to core ranking algorithms. The question was a long one, below is the relevant part of it:  “I was wondering about when these big announced core updates happen, are there some parts of the linking algorithm updated as well?” John Mueller began his answer by describing what the core algorithm is and how it’s made up of a lot of algorithms, not just one giant core algorithm. This is what Mueller said: “So in general, a core algorithm update, the way that we have it defined is somewhat vague. Because it’s not that we have this one piece of machinery that is the core algorithm and when we change one screw there, then that’s a core algorithm update.” Next, Mueller described what actually happens during a core algorithm up...
Read more

Subdomain vs. Subfolder: What Best Fits Your SEO Needs?

Subdomains and subfolders each offer different features and SEO benefits for your website. But, the age old question between search engine optimizers is which one is better for SEO rankings? Subdomains or subfolders? The answer often depends on the needs of your site, but it is also helpful to have a thorough understanding of each. Let’s break down the definitions of subdomains and subfolders for you and identify the advantages and disadvantages of each distinction. What Is a Subdomain? A subdomain is the section of a domain that is listed before the main domain name and the extension. For example, in the URL blog.bluleadz.com, blog is the subdomain. To begin understanding subdomains, you must understand the structure of a URL. A protocol, such as “https:” is a set of rules that allows data to be sent as requests to the web server. The domain, “bluleadz.com,” is the URL to the website’s main page.  Domains also have two parts, the top level domain (.com, .gov, .net, etc.) and the...
Read more