---
type: "Guide"
title: "Pre-rendering at scale — large sites with 100k+ pages"
description: "Operational playbook for pre-rendering sites with 100,000+ URLs: sitemap sharding, tiered TTL, selective pre-rendering, and render-pool cost control."
resource: "https://prerender.info/guides/large-sites-100k-pages"
tags: [guides, large site seo, 100k pages pre-rendering, sitemap sharding, tiered ttl, selective pre-rendering]
timestamp: 2026-04-21T00:00:00Z
---

# Pre-rendering at scale — large sites with 100k+ pages

Operational playbook for pre-rendering sites with 100,000+ URLs: sitemap sharding, tiered TTL, selective pre-rendering, and render-pool cost control.

## Selective pre-rendering

Selective pre-rendering is the practice of choosing which URL segments to pre-render based on traffic-delivery potential, while letting the rest remain client-rendered or canonical-redirected. It trades full coverage for render-pool cost control.

At 100,000+ URLs, pre-rendering everything is wasteful. The 80/20 rule holds: most organic traffic comes from a bounded canonical set, and the long tail is better served by canonical discipline than by rendering. If you are still proving the baseline problem, start with [crawl budget fundamentals](/guides/crawl-budget.md).

This guide is the operational playbook for that scale. Six steps from auditing the surface to tuning the render pool. It pairs directly with [JavaScript rendering cost](/guides/javascript-rendering-cost.md) and [ostr.io vs Cloudflare](/compare/vs-cloudflare.md) when the question becomes build vs buy at scale.

## Steps

1. **Audit the crawlable surface by segment** — Group URLs by template (products, categories, filters, seller-pages, blog, docs, etc.). Pull impressions per segment from Search Console. The top 3-5 segments usually cover 90% of traffic.
2. **Classify URLs by pre-render priority** — Tier A: top 20% canonicals with traffic or explicit priority (homepage, top categories, featured products). Tier B: long tail with any measurable impressions. Tier C: everything else (canonical-redirected, not pre-rendered).
3. **Shard the sitemap to match the tiers** — sitemap-a.xml contains Tier A with frequent lastmod. sitemap-b.xml contains Tier B with longer lastmod intervals. Tier C is not in the sitemap. Googlebot processes shards in parallel, so Tier A gets priority.
4. **Set tiered TTL on the pre-render pool** — Tier A: 1-4 hour TTL. Tier B: 24-72 hour TTL. Tier C: not pre-rendered. Render-pool cost scales with TTL × URL count; tiering keeps cost manageable.
### Per-prefix TTL configuration

```json
{
  "defaultTtlSeconds": 86400,
  "overrides": [
    { "prefix": "/",              "ttlSeconds": 3600 },
    { "prefix": "/category/",     "ttlSeconds": 7200 },
    { "prefix": "/p/",            "ttlSeconds": 14400 },
    { "prefix": "/blog/",         "ttlSeconds": 86400 },
    { "prefix": "/long-tail/",    "ttlSeconds": 259200 }
  ]
}
```

5. **Return rel=canonical on Tier C URLs** — Filter permutations, paginated deep URLs, and near-duplicates should canonical to the Tier A or Tier B parent. This removes them from the crawlable surface without hurting discoverability.
6. **Monitor render-pool utilisation weekly** — Render-pool utilisation should stay under 70% during peak crawl windows. If it climbs higher, either add render capacity or move the bottom of Tier B to Tier C. Cost scales linearly with utilisation.

## The real work is building the priority matrix

The mistake most teams make at 100k+ URLs is thinking in terms of page types only. The better model is a priority matrix: template type, impressions, update frequency, conversion value, and crawl cost. That matrix tells you what belongs in Tier A, B, and C.

If the site behaves more like a listing marketplace or aggregator than a classic catalogue, compare this guide with [travel aggregators](/use-cases/travel-aggregators.md) and [marketplaces](/use-cases/marketplaces.md).

## The operational threshold matters as much as the traffic threshold

Two teams can both have 150k URLs and still need different architectures. The difference is whether they can own invalidation, observability, and render-pool capacity themselves. That is why scale decisions must be read together with [headless browser overhead](/guides/advanced/headless-browser-overhead.md).

When the team cannot support that layer reliably, selective pre-rendering plus a managed service is usually cheaper than trying to optimize a perfect DIY system.

## FAQ

### How do we decide where the Tier B / Tier C line is?

Use the Search Console impression threshold. URLs with zero impressions over 30 days can usually be canonical-redirected. URLs with 1-10 impressions belong in Tier B. URLs with 10+ belong in Tier A.

### What about newly-published URLs?

New URLs enter Tier B by default and get promoted to Tier A if they accumulate impressions in the first month. Automate this with a weekly cron that re-reads Search Console data.

### Does pre-rendering slow down CI/CD on large sites?

No, because pre-rendering runs off-band from the deploy. The render pool processes snapshots asynchronously. Deploys remain fast; the snapshot refresh catches up in minutes.

### How do we handle 1M+ sites?

Above 1M URLs, selective pre-rendering is mandatory. Most teams pre-render 10-20% of URLs and let the remaining 80% canonical or noindex. Render-pool cost dominates at this scale.

## Related

- [Guide — crawl budget 80 percent](/guides/crawl-budget-80-percent.md) — Where budget gets lost on JS sites.
- [Guide — expand crawl budget](/guides/expand-crawl-budget.md) — Systematic levers.
- [Guide — cache headers](/guides/prerender-cache-headers.md) — TTL tuning per tier.
- [Use case — marketplaces](/use-cases/marketplaces.md) — Typical 1M+ URL pattern.
- [Use case — travel aggregators](/use-cases/travel-aggregators.md) — Deep fanout at 1M+ URLs.
- [Compare — vs Cloudflare](/compare/vs-cloudflare.md) — KV cost dominates at 100k+ URLs.

## Source

Generated from prerender.info editorial content (single source of truth). The canonical, always-current version is the `resource` URL above. Claims reflect prerender.info / ostr.io editorial guidance and are not confirmed from third-party public documentation unless explicitly linked.