What is crawl budget?

Crawl budget is the set of URLs that Googlebot will fetch from a site over a period of time. Google's crawler does not visit every page on every visit. It balances two things: how much crawling your server can handle without slowing down, and how often Google thinks the content is worth revisiting. The result is a practical ceiling on how many of your pages get looked at in a given window.

For most small and mid-sized sites, crawl budget is not a concern. Google typically finds and indexes a modest site without any special effort. It starts to matter when a site has a very large number of URLs, such as an e-commerce catalogue with many filtered or faceted listing pages, a property portal with thousands of near-identical listings, or a news or classifieds site that publishes constantly. In those cases, wasted crawling on duplicate pages, redirect chains, or endless parameter URLs can push new or updated pages further down the queue.

Crawl budget is also affected by site speed and server errors. If pages respond slowly or return frequent 5xx errors, Googlebot tends to slow its visits. Keeping sitemaps accurate, consolidating duplicate URLs, and blocking low-value sections in robots.txt are common ways site owners manage how crawling is spent. Malaysian sites that run on shared hosting or serve traffic mainly from one region can feel these limits sooner than they expect.

Crawl budget is a technical concept, so it usually falls under technical SEO work. Its importance depends on the size and structure of the site, and it is rarely a limiting factor for a simple brochure site.

"