---
type: "Guide"
title: "Expand your crawl budget — 7 levers that actually work"
description: "Seven engineering levers that expand crawl budget on JavaScript sites: sitemap sharding, canonical discipline, pre-rendering, cache headers, internal linking, response-code hygiene, and redirect cleanup."
resource: "https://prerender.info/guides/expand-crawl-budget"
tags: [guides, expand crawl budget, crawl budget optimization, crawl rate, googlebot efficiency, javascript seo]
timestamp: 2026-04-21T00:00:00Z
---

# Expand your crawl budget — 7 levers that actually work

Seven engineering levers that expand crawl budget on JavaScript sites: sitemap sharding, canonical discipline, pre-rendering, cache headers, internal linking, response-code hygiene, and redirect cleanup.

## Crawl budget expansion

Crawl budget expansion is the set of engineering actions that raise Googlebot's willingness to fetch and render URLs from a site, by proving host stability, removing waste, and signalling which URLs deserve priority.

If you are still building the mental model of how crawl budget works on JavaScript sites, start with [crawl budget fundamentals](/guides/crawl-budget.md) — this expansion guide assumes you already know which signals throttle versus reward crawl.

Crawl budget is not directly allocatable: you cannot ask Google for more. You can, however, remove signals that tell Googlebot to throttle your site and add signals that reward additional crawl.

The seven levers below are ordered by effort-to-impact. The first three move the needle on most sites within two sprints; the remaining four are long-tail improvements.

## Steps

1. **Shard the sitemap by priority** — Split one sitemap into three: fresh canonicals (daily lastmod), stable canonicals (weekly), and long tail (monthly). Googlebot processes shards in parallel, which effectively increases fan-out from your site.
2. **Enforce canonical discipline on filter URLs** — Return rel=canonical to the canonical filter URL for every filter permutation. This removes millions of URLs from the crawlable surface without hurting discoverability. The URL graph compresses to what actually ranks.
3. **Pre-render the canonical set** — Route crawler traffic on canonical URLs to a pre-rendered cache. Render queue depth drops, and Googlebot can crawl more URLs per cycle because each one is cheaper.
4. **Tune cache headers for recrawl efficiency** — Set Cache-Control max-age and Last-Modified so bot revalidation is cheap. Combined with pre-rendering, most recrawl hits become 304 Not Modified — the fastest possible response for the crawler.
### Response headers — bot-facing response

```text
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: public, max-age=3600, stale-while-revalidate=86400
Vary: User-Agent
Last-Modified: Tue, 21 Apr 2026 09:15:00 GMT
ETag: "snapshot-2d41f5a"
```

5. **Re-weight internal links on high-priority pages** — High-priority pages should link to other high-priority pages within 1-2 clicks. PageRank distributes through internal links; a page 4 clicks deep from the homepage is functionally invisible to crawlers at scale.
6. **Fix Soft 404s and 3xx redirect chains** — Search Console's Index Coverage report surfaces Soft 404s. Each one wastes a crawl slot. Redirect chains longer than 1 hop also burn budget. Cleaning both is mechanical but time-consuming.
7. **Remove client-side blocking on hydration** — Large JavaScript bundles that block rendering stall the render queue. Pre-rendering bypasses this for crawler traffic, but for non-pre-rendered pages, hydration performance still caps crawl rate. Audit the top 20 URL patterns for bundle size.

## FAQ

### How fast does crawl budget respond to these changes?

Sitemap sharding and canonical discipline produce visible change in 2-4 weeks. Pre-rendering and cache headers take 1-2 weeks. Internal linking changes propagate slowest — 4-12 weeks depending on link graph size.

### Should we request a manual crawl rate increase?

The old Crawl Rate setting in Search Console was removed. Google now adjusts automatically based on host load and site health. The levers above are the only way to signal "we can handle more."

### Does reducing page count help?

Yes, if the removed pages were zero-value. Removing Soft 404s and thin pages concentrates crawl on the remaining URLs. Do not remove pages that have any organic traffic or inbound links.

## Related

- [Guide — crawl budget fundamentals](/guides/crawl-budget.md) — Start here if the concept is new.
- [Guide — 80% crawl-budget loss](/guides/crawl-budget-80-percent.md) — Quantifies the render-queue waste on JS sites.
- [Guide — large sites 100k+](/guides/large-sites-100k-pages.md) — Scale-specific sharding strategies.
- [Guide — cache headers](/guides/prerender-cache-headers.md) — Cache-Control deep dive.
- [Guide — crawl frequency signals](/guides/crawl-frequency-signals.md) — Signals that raise recrawl rate.
- [Technology — how pre-rendering works](/technology/index.md) — Pipeline primer.

## Source

Generated from prerender.info editorial content (single source of truth). The canonical, always-current version is the `resource` URL above. Claims reflect prerender.info / ostr.io editorial guidance and are not confirmed from third-party public documentation unless explicitly linked.