Prerendering for Next.js: Setup Guide

Article

Implementing prerendering for a Next.js application requires four components: bot detection middleware, reverse proxy routing, WAF whitelist configuration, and snapshot validation. This guide covers all four with production-ready code for Next.js App Router (v13+) and ostr.io as the prerendering service.

ostr.io is a managed prerendering service that delivers deterministic HTML snapshots to search crawlers and AI retrieval systems through proxy-level middleware, without requiring framework rewrites. The middleware patterns below work with any prerendering service; ostr.io-specific configuration is noted where applicable.

Simple Integration: The seo-middleware-nextjs NPM Package

Before wiring bot detection by hand, consider the official NPM package: seo-middleware-nextjs. It is a lightweight Next.js middleware powered by the ostr.io pre-rendering engine, paired with a global CDN and smart caching layer. The package intercepts crawler and social-bot requests at the edge and reroutes them to ostr.io rendering endpoints — fully rendered HTML for bots, zero changes to the existing codebase, and less load on your origin (fewer server cycles and database queries).

It covers both traditional SERPs (Google, Bing, Yahoo) and modern AI agents (ChatGPT, Grok, Perplexity), plus link-preview bots from messaging and social apps (iMessage, Slack, Telegram, WhatsApp, Facebook, X/Twitter). Because pages are served pre-rendered from a CDN, it also improves TTFB, LCP, INP, CLS, and other Web Vitals — and with it, Lighthouse and SEO scores. PWAs, SPAs, styled-components, AMP, and strict Content-Security-Policy setups are supported. The package is written in strict TypeScript and exports all necessary types.

Install it:

1npm install seo-middleware-nextjs --save

Then create middleware.ts in /src/ or the root directory of your Next.js project:

1// middleware.ts
2import { SEOMiddleware } from 'seo-middleware-nextjs'
3
4const seoMiddleware = new SEOMiddleware({
5  // Get credentials from the ostr.io Pre-rendering Panel → Integration Guide
6  auth: 'Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
7})
8
9export const middleware = seoMiddleware.createMiddleware()
10
11export const config = {
12  matcher: '/((?!api|_next/static|_next/image|_next/webpack-hmr|\\.well-known|favicon.ico).*)',
13}

The auth token can also be supplied via the OSTRIO_AUTH, PRERENDER_SERVICE_AUTH, or SPIDERABLE_SERVICE_AUTH environment variable instead of hardcoding it. The constructor accepts options for custom bot lists (botAgents), excluded routes (ignoredPaths), ignore get parameters (keepGetQuery), and static-asset extensions (ignoredExtensions). See full documentation in seo-middleware-nextjs repository.

To speed up snapshot capture, signal readiness from your client bundle: initialize window.IS_RENDERED = false early, then set it to true once the page has finished loading data. See implementation example here.

If the package fits your deployment, — you don't need hand-written middleware in Step 2 and the Nginx routing in Step 3 — skip ahead to WAF whitelisting (Step 4), cache TTL (Step 5), and validation (Steps 7–9). The manual patterns below remain useful when you need full control over bot routing or run a non-standard proxy topology.

Step 1: Identify Which Routes Need Prerendering

Not every route in a Next.js App Router application needs prerendering. Server Components, Static Pages (with generateStaticParams), and ISR pages are already crawler-friendly. Prerendering adds value for:

• Routes using 'use client' components with data fetching (useEffect, React Query, SWR)
• Routes with significant client-side state that populates after mount
• Routes using Web Components or Shadow DOM content
• Any route where View Page Source shows significantly less content than the rendered DOM

Check each route category:

1# Find all client component files
2grep -r "'use client'" ./app --include="*.tsx" -l
3
4# Check which routes use client-side data fetching
5grep -r "useEffect\|useSWR\|useQuery" ./app --include="*.tsx" -l

Routes that are purely Server Components with no client-side data fetching can skip prerendering. Routes with the patterns above need it.

Step 2: Implement Bot Detection Middleware

Next.js middleware runs on every request at the Edge. The bot detection logic intercepts crawler requests and routes them to the prerendering service.

1// middleware.ts
2import { NextRequest, NextResponse } from 'next/server'
3
4const PRERENDERING_SERVICE_URL = process.env.PRERENDERING_SERVICE_URL ?? 'https://ostr.io'
5const PRERENDERING_TOKEN = process.env.PRERENDERING_API_TOKEN ?? ''
6
7// Bot User-Agent patterns — keep updated as new AI crawlers launch
8const BOT_UA_REGEX =
9  /googlebot|bingbot|yandexbot|duckduckbot|slurp|baiduspider|sogou|exabot|facebot|ia_archiver|gptbot|claudebot|applebot|anthropic-ai|perplexitybot|bytespider/i
10
11// Routes to exclude from prerendering
12const EXCLUDED_PATHS = [
13  /^\/_next\//,       // Next.js internal assets
14  /^\/api\//,         // API routes
15  /^\/static\//,      // Static files
16  /\.(js|css|png|jpg|jpeg|gif|svg|ico|woff|woff2|ttf|eot)$/i, // Assets
17]
18
19function isBot(userAgent: string): boolean {
20  return BOT_UA_REGEX.test(userAgent)
21}
22
23function isExcluded(pathname: string): boolean {
24  return EXCLUDED_PATHS.some((pattern) => pattern.test(pathname))
25}
26
27export function middleware(req: NextRequest) {
28  const ua = req.headers.get('user-agent') ?? ''
29  const { pathname } = req.nextUrl
30
31  // Skip non-bot traffic and excluded paths
32  if (!isBot(ua) || isExcluded(pathname)) {
33    return NextResponse.next()
34  }
35
36  // Build the prerendering request URL
37  const targetUrl = `${PRERENDERING_SERVICE_URL}/render/${req.url}`
38
39  // Forward to prerendering service with auth header
40  return NextResponse.rewrite(targetUrl, {
41    headers: {
42      'X-Prerender-Token': PRERENDERING_TOKEN,
43      'X-Forwarded-For': req.headers.get('x-forwarded-for') ?? req.ip ?? '',
44    },
45  })
46}
47
48export const config = {
49  matcher: [
50    // Match all routes except Next.js internals and static assets
51    '/((?!_next/static|_next/image|favicon.ico).*)',
52  ],
53}

Step 3: Configure Nginx Reverse Proxy (Alternative to Middleware)

If your deployment uses Nginx as a reverse proxy (common for non-Vercel deployments), configure bot routing at the Nginx layer:

1# /etc/nginx/conf.d/prerendering.conf
2
3mapmap $http_user_agent $is_bot {
4  default 0;
5  "~*googlebot"       1;
6  "~*bingbot"         1;
7  "~*yandexbot"       1;
8  "~*gptbot"          1;
9  "~*claudebot"       1;
10  "~*applebot"        1;
11  "~*anthropic-ai"    1;
12  "~*perplexitybot"   1;
13}
14
15serverserver {
16  listenlisten 443 ssl;
17  server_nameserver_name yourdomain.com;
18
19  locationlocation / {
20    # Route bots to prerendering service
21    ifif ($is_bot) {
22      proxy_passproxy_pass https://ostr.io/render/$scheme://$host$request_uri;
23      proxy_set_header X-Prerender-Token $PRERENDERING_TOKEN;
24      proxy_set_header X-Forwarded-For $remote_addr;
25      breakbreak;
26    }
27
28    # Route users to Next.js app
29    proxy_passproxy_pass http://localhost:3000;
30    proxy_set_header Host $host;
31    proxy_set_header X-Real-IP $remote_addr;
32  }
33}

Step 4: Add ostr.io IP Ranges to WAF Whitelist

Before prerendering traffic can reach your origin, ostr.io's IP ranges must be allowed in your WAF.

Cloudflare:

1. Go to Security → WAF → Tools → IP Access Rules
2. Add each ostr.io IP range with action: Allow
3. Note: Enter each CIDR block separately (Cloudflare does not support bulk import in the UI — use the API for large range lists)

1# Cloudflare API bulk allowlist
2curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/firewall/access_rules/rules" \
3  -H "Authorization: Bearer $CF_API_TOKEN" \
4  -H "Content-Type: application/json" \
5  -d '{"mode":"whitelist","configuration":{"target":"ip_range","value":"<ostr.io CIDR>"},"notes":"ostr.io prerendering"}'

AWS WAF:

1# Terraform: AWS WAF IP set for ostr.io
2resource "aws_wafv2_ip_set" "ostrio" {
3  name               = "ostr-io-prerendering"
4  scope              = "CLOUDFRONT"
5  ip_address_version = "IPV4"
6  addresses          = [/* ostr.io IP ranges */]
7}
8
9resource "aws_wafv2_web_acl_rule" "allow_ostrio" {
10  name     = "AllowOstrio"
11  priority = 1
12  action { allow {} }
13  statement {
14    ip_set_reference_statement {
15      arn = aws_wafv2_ip_set.ostrio.arn
16    }
17  }
18  visibility_config {
19    cloudwatch_metrics_enabled = true
20    metric_name                = "AllowOstrio"
21    sampled_requests_enabled   = true
22  }
23}

Count what this step costs you on an ongoing basis, because it is easy to file as a one-time task. The allowlist is a standing contract between two vendors who do not know about each other: your firewall provider classifies traffic, your rendering provider generates it, and you maintain the rule that stops the first from blocking the second. Ranges change, managed rule groups update, and a bot-detection toggle flipped by a colleague breaks the crawler path months later with no alert — the failure is silent by construction, which is why the WAF over-blocking guide exists at all.

There is an architecture where this step does not exist: an edge that owns both the firewall and the crawler-facing render, so no external range needs allowlisting because the rendering traffic never crosses a vendor boundary. Bridge CDN, from the same team as this site, is being built to that shape. It has not launched — no dashboard, no pricing, waitlist only — so it changes nothing about the setup you are doing today. Note it if an edge migration is on your roadmap; otherwise write the Terraform above and put a calendar reminder on re-verifying the ranges.

Step 5: Set Cache TTL, and Handle Freshness With Invalidation

Content types have different freshness requirements, but read your vendor's TTL model before you design around one. On ostr.io, TTL is set per host, not per route, and the documented range is 2 hours to 744 hours (cache docs). Earlier versions of this guide showed a per-pattern TTL config object with 15-minute product TTLs; that configuration does not exist on ostr.io and the sub-hour values are below its documented minimum.

What that means in practice: pick one host-level TTL that is safe for your slowest-changing content, then drive freshness for everything faster with event-driven invalidation rather than with a shorter clock.

1// Freshness comes from the publish event, not from a short TTL.
2// Host TTL: set once, conservatively (e.g. 24h) in the dashboard.
3// Per-URL refresh: invalidate on change, then re-request so the next
4// crawler hit is served from cache rather than waiting on a render.
5export async function onContentPublish(url: string) {
6  // 1. Invalidate the cached snapshot for this URL.
7  //    ostr.io keys per-page purges on the X-Prerender-Id response header;
8  //    a full purge is rate-limited to once per 2 hours, so never use it here.
9
10  // 2. Warm: fetch as a crawler so the snapshot is rebuilt now.
11  await fetch(url, {
12    headers: {
13      'User-Agent':
14        'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
15      'Cache-Control': 'no-cache',
16    },
17  })
18}

Which routes need an invalidation hook:

• Prices, inventory, real-time data — hook every write
• Product and listing pages — hook status and price changes
• Blog posts, docs, static content — hook publish only; the host TTL is enough
• Documentation, legal pages: 24 hours

Step 6: Warm Snapshots on Content Publish

Trigger snapshot regeneration on content publish events. For Next.js, the natural hook is after revalidatePath().

Note what this is not: ostr.io publishes no API endpoint for triggering a render, and earlier revisions of this guide showed a POST https://ostr.io/api/cache-warm call that does not exist. The working pattern is to request your own URL as a crawler, which makes the prerendering layer generate and cache the snapshot on your schedule instead of Googlebot's.

1// lib/prerendering.ts
2//
3// There is no documented public ostr.io endpoint for triggering a render.
4// Warming is therefore a fetch of your own URL with a crawler User-Agent:
5// the prerendering layer sees a crawler request, generates the snapshot, and
6// caches it — so the next real crawler visit is a cache hit rather than a wait.
7export async function warmSnapshot(url: string) {
8  try {
9    await fetch(url, {
10      headers: {
11        'User-Agent':
12          'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
13        'Cache-Control': 'no-cache',
14      },
15    })
16  } catch {
17    // Non-critical — log but don't throw
18    console.warn(`Snapshot warm failed for ${url}`)
19  }
20}
21
22// In your CMS webhook handler or Server Action
23import { revalidatePath } from 'next/cache'
24import { warmSnapshot } from '@/lib/prerendering'
25
26export async function publishContent(slug: string) {
27  const url = `https://yourdomain.com/blog/${slug}`
28
29  // Revalidate ISR cache
30  revalidatePath(`/blog/${slug}`)
31
32  // Warm the crawler-facing snapshot
33  await warmSnapshot(url)
34}

Step 7: Test the Prerendering Configuration

Before going live, verify that:

1. Bots receive prerendered HTML:

1# Simulate Googlebot request
2curl -A "Googlebot" https://yourdomain.com/blog/your-post -I
3
4# Expected: X-Prerender-Status: hit or X-Prerender-Status: rendered header
5# If missing: middleware is not routing bot traffic correctly

2. Users receive the live Next.js application:

1# Simulate user request (no bot UA)
2curl https://yourdomain.com/blog/your-post -I
3
4# Expected: no prerender headers, normal Next.js response

3. WAF is not blocking prerendering traffic: Check ostr.io dashboard for WAF-blocked render attempts. Check your WAF logs for blocked requests from ostr.io IP ranges.

4. DOM Consistency Score is above 95%: Use ostr.io's dashboard to compare the rendered snapshot for a representative product page against the live page. DOM Consistency Score should be 95%+.

5. Prerendering.info Checker: Submit your domain to the Prerender Checker and verify that the tool can retrieve a complete prerendered snapshot for your most important pages.

Step 8: Validate JSON-LD and Structured Data

Structured data (JSON-LD) must appear in the prerendered HTML, not be injected by client-side JavaScript. Verify with View Page Source (not DevTools, which shows the rendered DOM):

1# Check JSON-LD in prerendered output
2curl -A "Googlebot" https://yourdomain.com/product/your-product | grep -o 'application/ld+json' | wc -l
3# Should return 1 or more

If JSON-LD blocks are missing from the prerendered HTML, they are being generated client-side. Move them to Server Components.

Step 9: Monitor with Google Search Console

After going live, monitor these signals in Google Search Console:

• Crawl Stats → Render type: Should shift from "JavaScript" to "HTML" rendering
• Coverage → Not indexed: Should decrease over 4–8 weeks as snapshots reach Googlebot
• URL Inspection: For any page still not indexed, use URL Inspection to fetch and render — verify the prerendered output is complete

• ostr.io vs Vercel SSR

  The build-vs-buy call this guide assumes you settled.

• Vercel ISR vs a prerendering service

  Why ISR does not cover the crawler path on its own.

• Guide - pre-render cache headers

  Cache-Control and Vary once middleware is live.