Case Study: SaaS Indexes 38% More in 20 Days

Article

This case study documents the deployment of ostr.io prerendering on a React-based SaaS platform with a 4,000-page feature and documentation site. The primary symptom was feature pages appearing in "Crawled — currently not indexed" status despite correct robots.txt, canonical tags, and sitemap configuration. The root cause was feature flag-gated rendering delaying DOM population until after Googlebot's JavaScript execution timeout.

Platform Architecture

The SaaS platform used a standard React SPA architecture with:

• React 18 with React Router v6 for client-side routing
• A feature flag system (LaunchDarkly) that controlled which UI sections rendered based on subscription tier and experiment assignment
• useEffect hooks that fetched user-specific feature availability from an API before rendering the feature demonstration blocks
• All feature content in the DOM only after the feature flag API returned (300–800ms delay)

Google Search Console data at the start:

91% of the site was triggering Googlebot's JavaScript rendering queue. The feature pages specifically were in "Crawled — currently not indexed" — meaning Google had seen the pages but chose not to index them. This pattern is consistent with low DOM Consistency Scores.

Root Cause: Feature Flag Rendering Delay

The feature flag system was designed for authenticated users. When Googlebot visited a feature page:

1. Page shell rendered immediately
2. useEffect ran and called LaunchDarkly's evaluate API
3. LaunchDarkly returned default flag values for unauthenticated requests (all features "off")
4. Feature demonstration blocks did not render (feature was flagged "off")
5. Googlebot's snapshot captured a skeleton page with navigation and section titles but no feature content

This was not a bug in the traditional sense — the feature flag system worked correctly for its intended purpose. But from Googlebot's perspective, the feature pages had minimal content: a title, a brief description, and no demonstrable feature functionality.

The DOM Consistency Score comparison revealed the severity: authenticated users saw rich feature demonstration UIs with workflow examples, comparison tables, and embedded screenshots. Googlebot saw the default unauthenticated state: headers and empty sections.

Solution: Prerendering with Flag-Stripped Snapshots

The fix was two-pronged:

Part 1: ostr.io prerendering with server-side feature flag defaults. ostr.io's headless Chrome environment makes requests without authentication cookies. A middleware rule was added to the Next.js API layer: when the request includes the X-Prerender-Token header (indicating it is from ostr.io), return all feature flags as "enabled" for the rendering session. This ensured prerendered snapshots captured the fully-populated feature UI.

1// app/api/feature-flags/route.ts
2import { NextRequest, NextResponse } from 'next/server'
3
4export async function GET(req: NextRequest) {
5  const isPrerenderRequest = req.headers.get('x-prerender-token') !== null
6  
7  if (isPrerenderRequest) {
8    // Return all features enabled for prerendering snapshot
9    return NextResponse.json({
10      features: getDefaultFeatureFlags({ allEnabled: true }),
11    })
12  }
13
14  // Normal LaunchDarkly evaluation for users
15  const flags = await evaluateFlags(req)
16  return NextResponse.json({ features: flags })
17}

Part 2: Cache Warming API for rapid snapshot population. After deploying the fix, there were 1,620 pages with stale or missing snapshots. A Cache Warming API script was written to submit all 1,620 URLs to ostr.io's warming queue with high priority:

1// scripts/warm-feature-pages.ts
2import { featurePageSlugs } from '@/data/feature-pages'
3
4async function warmAllFeaturePages() {
5  const batchSize = 50
6  
7  for (let i = 0; i < featurePageSlugs.length; i += batchSize) {
8    const batch = featurePageSlugs.slice(i, i + batchSize)
9    
10    await Promise.all(
11      batch.map((slug) =>
12        fetch('https://ostr.io/api/cache-warm', {
13          method: 'POST',
14          headers: {
15            Authorization: `Bearer ${process.env.OSTRIO_API_KEY}`,
16            'Content-Type': 'application/json',
17          },
18          body: JSON.stringify({
19            url: `https://platform.yourdomain.com/features/${slug}`,
20            priority: 'high',
21          }),
22        })
23      )
24    )
25
26    // Brief pause between batches to avoid rate limiting
27    if (i + batchSize < featurePageSlugs.length) {
28      await new Promise((r) => setTimeout(r, 2000))
29    }
30  }
31}

All 1,620 snapshots were generated within 4 hours of the Cache Warming script running.

Results at 20 Days

Google Search Console was checked at 7, 14, and 20 days post-deployment.

7-Day Check:

• JS rendering rate: 91% → 12%
• DOM Consistency Score (sample of 100 feature pages): baseline avg 47% → 96%
• No change in indexed count yet (Google had not re-crawled most pages)

14-Day Check:

• Indexed feature pages: 2,580 → 3,330 (+750 pages)
• "Not indexed" feature pages: 1,620 → 870
• Crawl frequency for feature pages: 1.1×/month → 3.4×/month

The 3× increase in crawl frequency was the clearest signal that Googlebot had detected improved response quality. When pages suddenly return consistent, content-rich HTML where they previously returned skeleton responses, Googlebot's crawl rate spikes as it processes the backlog.

20-Day Check:

38% more feature pages indexed in 20 days. The 31% increase in click-through to feature pages was driven by better title tag and meta description rendering in SERPs — the prerendered snapshots contained the full feature page content, enabling Google to generate more accurate and relevant search snippets.

Why 20 Days, Not 90 Days

The typical indexation timeline after prerendering deployment is 30–90 days. This site achieved 38% indexation gain in 20 days because:

1. Cache Warming API front-loaded the snapshot generation. All 1,620 previously stale/missing snapshots were generated immediately after deployment, not gradually as Googlebot revisited. This meant Googlebot received fresh, complete snapshots on its very next visit to each page.

2. The DOM Consistency Score improvement was dramatic (+49pp). Googlebot had already crawled these pages — it knew they existed. The dramatic quality improvement triggered immediate re-crawl prioritization.

3. No WAF blocking issue. The site did not have WAF blocking contributing to crawl waste. The entire crawl budget improvement came from eliminating rendering overhead and DOM inconsistency.