Crawl budget: what it is and how to optimise it | Ighenatt

Every time Googlebot visits your site, it has limited time. It is not going to crawl all your pages on every visit; it has a budget. If that budget runs out traversing duplicate pages, internal search filters or URLs with irrelevant parameters, your most important pages may go uncrawled for weeks.

For small sites, this is rarely a problem. But if you manage an ecommerce store with thousands of products, a content portal with hundreds of articles or a site with programmatic page generation, the crawl budget determines how much of your content actually reaches Google’s index.

What crawl budget is and why it matters

Crawl budget is the combination of two factors Google uses to decide how many pages it crawls from your site in a given period:

Crawl rate limit: the maximum speed at which Googlebot can crawl your site without degrading the user experience. If your server responds slowly or returns 5xx errors, Google automatically reduces crawl frequency to avoid overloading it.

Crawl demand: Google’s interest in your URLs. Popular, frequently linked or recently updated pages receive more crawl demand. Pages with low traffic, duplicate content or low perceived quality receive less attention.

The result is that Google crawls a finite number of pages per visit. If your site has 100,000 URLs but Googlebot only crawls 5,000 per day, the pages that do not make it into that daily selection wait. If those excluded pages are your star products or your most recent content, you are losing indexation opportunities and, by extension, organic traffic.

Understanding this mechanism is a fundamental part of technical SEO for any site with a significant volume of pages.

Factors that influence your site’s crawl budget

Several elements determine how Google allocates its crawl budget:

Server speed. A server that responds in under 200ms allows Googlebot to crawl more pages per second. A server that takes more than 1 second per response drastically reduces the total number of pages crawled per session.

URL structure. URLs with tracking parameters, combinable filters and sorting variants can multiply the number of URLs perceived by Google. A catalogue of 1,000 products can generate 50,000 URLs if each filter combination creates a different URL.

Update frequency. Pages that change frequently attract more crawling. Static pages that have not been modified for months receive fewer Googlebot visits.

Internal and external links. Well-linked pages (both internally and from other sites) receive more attention from Googlebot. An orphan page (with no internal links pointing to it) may take much longer to be crawled.

HTTP errors. A high percentage of 404 or 5xx errors during crawling reduces Google’s confidence in the site’s technical quality, which can decrease the crawl rate.

Signs that your crawl budget is poorly optimised

Detecting crawl budget problems requires observing patterns, not isolated events:

• New or updated pages that take weeks to appear in Google’s results.
• The Search Console coverage report shows a growing number of URLs “Discovered — currently not indexed”.
• Crawl statistics show Googlebot spending time on URLs that provide no SEO value (filter pages, infinite pagination, session parameters).
• The ratio between pages crawled and pages indexed is low (below 60%).
• URLs with JavaScript that blocks rendering consume crawl budget but deliver no indexable content.

How to view crawl budget in Google Search Console

Google Search Console offers a crawl statistics report accessible from Settings > Crawl stats. This report shows:

Total crawl requests: how many pages Googlebot has requested per day. A downward graph can indicate server problems or Google’s loss of interest in the site.

Total download size: how much data Googlebot downloads. If this number is high but requests are low, your pages are heavy and Googlebot is spending bandwidth instead of crawling more URLs.

Average response time: how long your server takes to respond. Values above 500ms are an indicator that the server is limiting the crawl rate.

Breakdown by response type: what percentage of requests receive 200, 301, 404 or other codes. A significant percentage of 3xx or 4xx indicates that Googlebot is wasting budget on URLs that do not deliver useful content.

Breakdown by file type: what proportion of crawling is devoted to HTML, JavaScript, CSS, images. If Googlebot devotes an excessive portion to crawling static files, it may be neglecting your content pages.

7 techniques to optimise your crawl budget

1. Configure robots.txt to block sections with no SEO value. Internal search filters, sorting pages, admin areas and tracking URLs do not need to be crawled. Use Disallow in robots.txt to exclude them.

2. Manage URL parameters. If your site generates URL variants with parameters (?color=red&size=m), implement canonical tags towards the main URL and consider using robots.txt to block filter combinations.

3. Keep sitemap.xml updated and clean. Include only URLs you want indexed: pages with unique content, 200 response code and no noindex tag. Remove obsolete, redirected or duplicate URLs.

4. Improve server speed. A TTFB below 200ms allows Googlebot to crawl more pages per session. Invest in caching, CDN and database optimisation before thinking about more advanced configurations.

5. Implement an efficient internal link structure. Important pages must be at most 3 clicks from the home page. Internal links distribute crawl authority and guide Googlebot towards priority content.

6. Eliminate redirect chains. Each redirect consumes one crawl budget request. Chains of 3 or more redirects multiply waste. Always point to the final destination with a single 301 redirect.

7. Consolidate duplicate content. If you have several URLs with similar content (www and non-www versions, HTTP and HTTPS, with and without trailing slash), use canonical tags and redirects to consolidate them into a single URL per piece of content.

Frequently asked questions about crawl budget

Does crawl budget affect small sites?

Generally not. Google has confirmed that sites with fewer than a few thousand URLs do not have crawl budget problems. The server responds quickly and Google can crawl all pages without restriction. Crawl budget becomes a critical factor from 50,000 URLs upwards.

How do I block low-value pages from Google’s crawl?

Use robots.txt to block entire sections that provide no SEO value (such as internal filters, internal search results or admin areas). For individual pages, use the noindex meta tag if you want Google not to show them but still crawl them, or robots.txt if you want to prevent crawling completely.

What is crawl demand and crawl rate?

The crawl rate limit is the maximum speed at which Googlebot can crawl without overloading your server. Crawl demand is Google’s interest in crawling your URLs, based on popularity, freshness and content quality. Both factors combined determine your effective crawl budget.

If you suspect Googlebot is not crawling your most important pages or that your crawl budget is being wasted on irrelevant URLs, you need a technical diagnosis. Contact our team to audit your site’s crawling and design a strategy that directs Googlebot towards the pages that really matter.