What is Crawl Budget?<\/h2>\n
Crawl budget is not a single number but a general term that describes how often and how many pages Google crawls and indexes from a particular website over a given period of time.<\/p>\n
Factors affecting crawl budget are website and navigation structure, duplicate content (within the site), soft 404 errors, low-value pages, website speed and hacking issues.<\/p>\n
Why Is Crawl Budget Important for SEO?<\/h2>\n
It should be emphasized from the beginning that crawling is not a ranking signal.<\/p>\n
This means that crawling does not directly impact the position a page will appear in organic search results<\/a><\/strong>.<\/p>\n But crawling and crawl budgets are important for SEO because:<\/p>\n What is crawl budget optimization?<\/strong><\/p>\n Crawl budget optimization is the process of checking and making sure that search engines can crawl and index all important pages of your site on time.<\/p>\n Crawl budget optimization is not usually an issue with small websites but it\u2019s more important for big websites that have thousands of URLs.<\/p>\n Nevertheless, as you will read below, the way to optimize your crawl budget is by following SEO best practices<\/a><\/strong> and this has a positive effect on your rankings too.<\/p>\n Follow the 10 tips below to optimize the crawl budget for SEO.<\/p>\n When search engine crawlers visit a site, they start from the homepage and then follow any links to discover, crawl and index all website pages.<\/p>\n Having a hierarchical site structure with no more than 3 levels deep is the ideal structure of any kind of website.<\/p>\n This means that any URL should be accessible from the homepage in three clicks or less.<\/p>\n This simple structure makes crawling easier and faster and it\u2019s good for the users too.<\/p>\n For any type of website, search engines like to give more priority (when it comes to crawling and indexing) to the most important pages of a site.<\/p>\n One of the ways they use to identify the important pages of a site is the number of external and internal links a webpage has.<\/p>\n External links are more important but harder to get, while it\u2019 easy for any webmaster to optimize their internal links.<\/p>\n Optimizing internal links in a way that it helps crawl budget means:<\/p>\n Having pages on your site that have no internal or external links (also called \u2018orphan pages\u2019) makes the job of search engine bots more difficult and they waste your crawl budget.<\/p>\n Speed is an important ranking factor<\/a><\/strong>, a great usability factor and a factor that affects the crawl budget.<\/p>\n To say it simply, when a website loads fast the Googlebot can crawl more pages of the same site in less time. This is a sign of a healthy website infrastructure and an encouragement to crawlers to get more content form the particular site.<\/p>\n This is what Google mentions<\/a><\/strong> about site speed and crawl budget.<\/p>\n As a webmaster, your job is to make every effort to ensure that your webpages load as fast as possible on all devices.<\/p>\n One of the factors that can negatively impact crawl budget is on-site duplicate content.<\/p>\n Duplicate content in this context<\/em> is identical or very similar content appearing in more than one URL on your site.<\/p>\n This is a very common issue in eCommerce category pages where similar products are listed in more than one category.<\/p>\n Besides eCommerce sites, blogs can have issues with duplicate content. For example, if you have a number of pages targeting the same keywords, and the content on those pages is similar, then Google may consider this as a duplicate content.<\/p>\n How does duplicate content impacts crawl budget?<\/strong><\/p>\n It makes the job of Googlebot more difficult because it has to decide which of the pages to index.<\/p>\n Crawling resources gets wasted on pages that Google will eventually mark as a duplicate content.<\/p>\n Pages that are more valuable to the site may not get indexed because the crawl rate limit might have been reached crawling and indexing duplicate content pages.<\/p>\n How to solve duplicate content issues?<\/strong><\/p>\n The best way to solve duplicate content issues is to:<\/p>\n Similar to duplicate content, another factor that can impact crawl budget is thin content pages.<\/p>\n Thin content is pages on your site that have little or no content and add no value to the user. They are also referred to as low-quality pages or low-value pages.<\/p>\n Examples are pages that have no text content, empty pages or old published pages that are no longer relevant to both search engines and users.<\/p>\n To optimize your crawl budget you should find and fix thin content pages<\/a><\/strong> by:<\/p>\n By doing any of the above actions, crawling time will be allocated on pages that are important for your site.<\/p>\n Soft 404 errors<\/a><\/strong> can happen for many reasons and it\u2019s not always easy to find out the exact reason.<\/p>\n The most common is the misconfiguration of your HTTP server, slow loading websites and having a lot of thin content pages on your site.<\/p>\n The problem with soft 404 errors (in comparison to normal 404 errors), is that soft 404 errors waste your crawl budget because search engine crawlers keep these pages in their index and try to recrawl them.<\/p>\n The best way to deal with soft 404 errors and optimize your crawl budget is to log in to Google search console<\/a><\/strong> and view the Coverage error report.<\/p>\n Click on \u201cSubmitted URL seems to be a Soft 404\u201d to view the list of affected pages and fix them.<\/p>\n Another way to increase your crawl budget is to reduce the number of crawl errors. Crawling time spend on errors that shouldn\u2019t exist in the first place is wasted time.<\/p>\n The easiest way to do this is to use the Google search console \u201cIndex Coverage Report\u201d, to find and fix crawl errors.<\/p>\n Our comprehensive guide \u201cHow to fix crawl errors in Google search console<\/a><\/strong>\u201d has all the information you need.<\/p>\n Another issue that may slow down how often Google crawls a website is the presence of too many redirects.<\/p>\n Redirects are a great way to solve duplicate content issues and soft 404 errors, but care should be taken not to create redirect chains.<\/p>\n When the Googlebot finds a 301 redirect, it may not crawl the redirected URL immediately but will add to the list of URLs to crawl from the particular site. If a URL is redirected to a URL and that URL is redirected to a new URL then this complicates the process and slows down crawling.<\/p>\n Check your .htaccess and make sure that you don\u2019t have any unnecessary redirects and that any 301 redirects are pointing to the final destination only (avoid intermediate destinations for the same URL).<\/p>\n A website that is hacked has a lot more things to worry about than crawl budget, but you should know how hacked pages affect crawl budget.<\/p>\n If your website is hacked for some time without you knowing about it, this will result in reducing your crawl budget considerably. Google will lose the trust of the site and index it less often.<\/p>\n To avoid this unpleasant situation, you can make use of a security service to monitor your website and check regularly the \u201cSecurity Issues\u201d report of Google search console (located under Security and Manual actions).<\/p>\n Popular URLs tend to be crawled more often by search engines because they want to keep their content fresh in their index.<\/p>\n In the SEO world, the biggest factor that differentiates popular pages from the least popular pages is the number and type of backlinks<\/a><\/strong>.<\/p>\n Backlinks help in establishing trust with search engines, improve a page\u2019s PageRank and authority and this eventually results in higher rankings.<\/p>\n It\u2019s one of the fundamental SEO concepts<\/a><\/strong> that hasn\u2019t changed for years.<\/p>\n So, having pages with links from other websites will encourage search engines to visit these pages more often resulting in an increase of crawl budget.<\/p>\n Getting links from other websites is not easy, in fact, it\u2019s one of the most difficult aspects of SEO but doing so will make your domain stronger and improve your overall SEO.<\/p>\n Although you should not get over obsessed about crawling budgets and crawl stats, it\u2019s good from time to time to review the \u201cCrawl Stats\u201d report in Google search console and look for any abnormal behavior.<\/p>\n The Crawl Stats report is currently available in the old version of the Google search console. To find it you need to login to your Google search console account and then select CRAWL STATS under \u201cLegacy Tools and Reports\u201d.<\/p>\n What this report shows is information about ALL Googlebot activity on your site for the last 90 days.<\/p>\n The report will include any attempt made by Googlebot to access any crawlable asset on your sites such as pages, posts, images, CSS files, js files, pdf emails and anything else that you have uploaded on your server.<\/p>\n That\u2019s also the reason why the number of \u201cPages crawled per day\u201d is bigger than the number of pages you have in Google index.<\/p>\n What to look for in the Crawl stats report?<\/strong><\/p>\n When viewing the report try to spot any sudden drops or spikes in the number of pages crawled by day. Look for a period of two weeks or a month and see if the drop or spike is continuous.<\/p>\n Under normal circumstances, the number of crawl pages should steadily increase over time (provided that you add new content on the site on a regular basis). If you are not making any changes, then the pattern should be similar if you compare two time periods.<\/p>\n A sudden drop in crawl rate can occur when:<\/p>\n A crawl rate can spike when:<\/p>\n You can visit the link below to learn more about the data shown in the crawl stats report but in the majority of cases, this is not something you should worry too much about.<\/p>\n Optimizing your crawl budget for SEO is the same process as optimizing your website for technical SEO<\/a><\/strong>. Anything you can make to improve your website\u2019s usability and accessibility is good for your crawl budget, is good for users and it\u2019s good for SEO.<\/p>\n Nevertheless, every little step helps SEO and when it comes to crawl budget optimization the most important step is to get rid of crawling and indexing errors. These errors waste your crawling budget and fixing them will contribute to your website\u2019s overall health.<\/p>\n The other factors like website speed, duplicate content and external links, can improve site visibility in search engines<\/a> <\/strong>and this means higher rankings and more organic traffic<\/a>.<\/strong><\/p>\n Finally, it\u2019s a good practice to take a lot at your crawl stats report from time to time to spot and investigate any sudden drop or spike in crawl rate.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":" Understanding what the meaning of a crawl budget is and how to optimize it for SEO purposes is important if you want to have full control of what is going on with your website. It\u2019s a highly technical subject and that\u2019s why most webmasters tend to avoid it, but it doesn\u2019t have to be this […]<\/p>\n","protected":false},"author":2,"featured_media":16693,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","mc4wp_mailchimp_campaign":[]},"categories":[3],"tags":[601,602],"_links":{"self":[{"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/posts\/16692"}],"collection":[{"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/comments?post=16692"}],"version-history":[{"count":10,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/posts\/16692\/revisions"}],"predecessor-version":[{"id":29271,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/posts\/16692\/revisions\/29271"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/media\/16693"}],"wp:attachment":[{"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/media?parent=16692"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/categories?post=16692"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.reliablesoft.net\/wp-json\/wp\/v2\/tags?post=16692"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
How to Optimize Your Crawl Budget for SEO<\/h2>\n
\n
1. Provide for a hierarchical website structure<\/h3>\n
![\"Ideal](\"https:\/\/www.reliablesoft.net\/wp-content\/uploads\/2018\/05\/ideal-structure-small-business.jpg\")
2. Optimize Internal linking<\/h3>\n
\n
\nInternal linking SEO best practices<\/a><\/strong> \u2013 A comprehensive guide on how to optimize your internal link structure.<\/div>\n3. Improve your Website speed<\/h3>\n
![\"Crawl](\"https:\/\/www.reliablesoft.net\/wp-content\/uploads\/2020\/01\/site-speed-crawl-budget.png\")
4. Solve Duplicate content issues<\/h3>\n
![\"Canonical](\"https:\/\/www.reliablesoft.net\/wp-content\/uploads\/2019\/10\/canonical-url.jpg\")
\n
5. Get rid of thin content<\/h3>\n
\n
6. Fix Soft 404 errors<\/h3>\n
![\"Find](\"https:\/\/www.reliablesoft.net\/wp-content\/uploads\/2020\/01\/seems-to-be-soft-404.png\")
\nHow to find and fix soft 404 errors<\/a><\/strong> \u2013 a step-by-step guide on how to identify soft 404s and possible ways to fix them.<\/div>\n7. Fix Crawl errors<\/h3>\n
![\"Google](\"https:\/\/www.reliablesoft.net\/wp-content\/uploads\/2019\/10\/search-console-index-coverage-report.jpg\")
8. Avoid having too many redirects<\/h3>\n
9. Make sure that you have no hacked pages<\/h3>\n
10. Improve your website\u2019s reputation (External links)<\/h3>\n
\nLink Building Techniques<\/a><\/strong> \u2013 11 link building techniques that actually work.<\/div>\nHow to Check and Interpret your Crawl Stats Report?<\/h2>\n
![\"Crawl](\"https:\/\/www.reliablesoft.net\/wp-content\/uploads\/2020\/01\/crawl-stats-report.png\")
\n
\n
\nCrawl Stats report<\/a><\/strong> \u2013 A guide from Google on how to correctly interpret the data in the crawl stats report.<\/div>\nKey Learnings<\/h2>\n