Case Studies
Case Study: Marketplace Kills navDemotion
How a property marketplace with 3.2M listings hit 99%+ DOM consistency and removed navDemotion rank instability.
Article
How to read this case study. The figures below are a composite, modelled on production rollouts rather than transcribed from one named client, and no client is identified because none consented to be. Treat the mechanism as the transferable part — the failure mode, the diagnosis order, and the sequence of fixes — and treat the numbers as illustrative of scale, not as a benchmark to expect. Anything you would put in a business case should come from your own Search Console and server logs.
This case study documents the prerendering implementation for a property marketplace with 3.2 million active listings. The marketplace used a React-based listing detail page architecture where listing data was fetched client-side via GraphQL after initial mount. The primary symptom was navDemotion on 23% of high-value listing pages — observable as ranking instability and periodic position drops for otherwise well-optimized pages.
ostr.io was deployed as the prerendering service, reducing DOM Consistency Score from 61% to 99.4% across the listing catalog and eliminating navDemotion signals within 45 days.
The navDemotion Problem
What is actually known about navDemotion. navDemotion is an attribute name that appeared in the Google Content Warehouse API documentation leak of May 2024, alongside other demotion fields such as anchorMismatchDemotion and serpDemotion. The leak exposed field names and, in some cases, one-line descriptions; it did not explain what computes them, what thresholds apply, or whether the field is still in use. Google has never documented navDemotion publicly and has not confirmed the leaked corpus reflects current ranking systems.
Everything beyond the name is inference. The reading used on this site — that the field relates to divergence between the DOM Googlebot indexes and the DOM users experience after JavaScript executes — is our interpretation, not an established fact, and the field name itself suggests navigation-related signals at least as readily as rendering parity. Treat it as a working hypothesis for why crawler/user DOM divergence might carry a ranking cost, and treat the mechanism below as the part that is directly observable: what Googlebot receives, what users receive, and whether the two differ. That gap is measurable regardless of which internal signal, if any, responds to it.
With that caveat in place: the pattern the marketplace exhibited was pages with strong independent signals (authority, backlinks, engagement) ranking well for weeks, then losing 5–15 positions suddenly without any content change.
The DOM inconsistency on this marketplace had a specific root cause: the React listing page component fetched listing details (price, availability, property attributes) via a GraphQL query in a useEffect hook. The server returned an HTML shell — navigation, layout, SEO metadata — but not the listing content. Googlebot's second-wave JavaScript execution populated the listing content, but with a time delay.
The crawl timeline for a typical listing page:
- Googlebot requests the listing URL
- Receives HTML shell (navigation + empty
<div id="listing-content"></div>) - Second-wave rendering executes the React application
- GraphQL query fires and returns listing data
- React renders the listing details into
#listing-content - Google captures the final DOM: listing content present
But the captured DOM sometimes differed from what users saw because:
- GraphQL query timing varied between Googlebot's execution environment and user browsers
- Feature flags controlling listing page UI elements produced different results in Googlebot's context vs. authenticated user sessions
- A/B tests on listing UI components were not excluded from the crawler path
The result: Googlebot indexed a listing page with Content A; users consistently saw Content B. Nobody outside Google can observe a demotion field firing — what was observable was the divergence itself, and the ranking instability that tracked it.
Diagnosis: DOM Consistency Score Analysis
Using ostr.io's monitoring tool (deployed in assessment mode before full rollout), 500 listing pages were sampled:
- 100 pages in "ranked position 1–10" (stable, high-performing listings)
- 200 pages in "ranked position 11–50" (typical mid-tier listings)
- 200 pages in "Crawled — currently not indexed" (worst-performing listings)
DOM Consistency Score results:
| Group | Avg crawler/live DOM match | Interpretation |
|---|---|---|
| Ranked 1–10 | 89% | Indexed document broadly represents the page |
| Ranked 11–50 | 61% | Roughly a third of the page missing from the indexed copy |
| Not indexed | 41% | Indexed copy is mostly shell |
The association was consistent: listing pages whose crawler-visible DOM closely matched the live page ranked well; those that diverged ranked poorly or were not indexed. Note the direction of the evidence — this is a correlation observed across one sample, not a demonstration that parity causes ranking. Pages that render fully for crawlers also tend to differ in other ways. The mid-tier 61% average is best read as a description of what Google was actually indexing for those URLs: roughly a third of the listing content was absent from the copy competing in search.
Implementation
Week 1: Prerendering middleware deployment and WAF configuration.
ostr.io's middleware was deployed via Cloudflare Worker in front of the marketplace's origin infrastructure. The marketplace used Cloudflare Enterprise with Bot Management — this required careful WAF rule configuration to ensure ostr.io's IPs were allowed before Bot Management rules evaluated them.
The WAF rule order:
- Allow: ostr.io IP ranges (priority 1 — execute before all other rules)
- Allow: Known Googlebot IP ranges (priority 2)
- Bot Management: existing configuration (unchanged)
Without this rule ordering, ostr.io render requests would have been blocked by Bot Management's behavioral analysis before the allow rule could apply.
Week 1–2: A/B test exclusion from crawler path.
The A/B testing framework (Optimizely) was configured to return control group content when the request included the X-Prerender-Token header from ostr.io. This ensured prerendered snapshots showed consistent, non-experimental UI.
Week 2: Feature flag normalization for crawler path.
LaunchDarkly was configured to return production-default flag values for prerendering requests. This eliminated the feature flag variance contributing to DOM inconsistency.
Week 2–3: GraphQL pre-fetching for listings.
The highest-impact change: listing GraphQL queries were moved from useEffect (client-side, after mount) to the Next.js App Router page.tsx using async/await server-side data fetching. This pre-populated listing content in the server response, eliminating the client-side GraphQL delay from the snapshot timeline.
This change required refactoring 23 listing page components across different property types (residential, commercial, rental, new development). The refactor took 2 engineers × 3 weeks.
Week 3–4: cache rebuild across 3.2M listings.
After the code changes were deployed, the existing snapshots were stale by definition — they had been generated against the old client-side-fetch architecture. Rebuilding them meant invalidating and re-requesting in priority order: listings updated in the last 7 days first, then listings with status: available, then the archive. The rebuild took about 4 days at the concurrency the account allowed.
Two practical constraints shaped this, and they apply to any ostr.io deployment. Cache TTL is set per host with a documented range of 2 to 744 hours, so a 30-minute TTL for fast-moving listings is not available — publish-triggered invalidation has to do that work instead. And a full purge is rate-limited to once per two hours, which is why a rebuild of this size runs as per-URL invalidation rather than a single purge-everything step.
Results
45-Day Metrics:
| Metric | Baseline | 45 Days | Change |
|---|---|---|---|
| Avg DOM Consistency Score | 61% | 99.4% | +38.4pp |
| Sampled URLs with material crawler/live divergence | 23% of top listings | <1% | -96% |
| Indexed listing pages | 1.9M | 2.7M | +42% |
| Wasted crawls (re-crawls of unchanged pages) | 67% of budget | 22% | -67pp |
| Avg crawl frequency (active listings) | 0.8×/month | 2.4×/month | +200% |
| Organic traffic (GSC, 28-day comparison) | Baseline | +34% | — |
The 67% reduction in wasted crawls was the most significant operational improvement. Before prerendering, Googlebot was re-crawling listings repeatedly because each visit produced inconsistent DOM output. With DOM Consistency Score at 99.4%, Googlebot's crawl model recognized that re-crawling these pages rarely produced different content — and reallocated that budget to crawling new, uncrawled listings.
The 3× increase in crawl frequency for active listings reflected Googlebot's prioritization of freshness for a site now delivering consistent, high-quality responses.
The 42% increase in indexed listings (from 1.9M to 2.7M) represented 800,000 additional property listings entering the index. For a property marketplace, each indexed listing is a potential landing page for location-specific search queries. The long-tail organic traffic gain from 800k new indexed listings drove 34% organic traffic growth over the 45-day measurement period.
Key Lessons
navDemotion is subtle but traceable. The ranking instability pattern (good rankings → sudden drops → recovery → repeat) is characteristic of navDemotion cycling. If you observe this pattern on otherwise well-optimized pages, DOM Consistency Score analysis is the first diagnostic step.
A/B tests and feature flags are the most common hidden DOM inconsistency sources. In this case, both were contributing factors. Excluding the crawler path from A/B tests and feature flag experiments is not optional — it is a prerequisite for achieving high DOM Consistency Scores on dynamic applications.
GraphQL client-side fetching at scale causes indexation failure at scale. Moving GraphQL queries to server-side execution (Next.js Server Components, getServerSideProps, or equivalent) is the architectural fix. Prerendering provides the bridge while the refactor is in progress — or permanently if the refactor is not feasible.
Frequently Asked Questions
You measure the cause rather than the effect, because the effect is not observable. Run a script that fetches the same 200 listing URLs three times each over 24 hours from a verified Googlebot-equivalent client, hash the rendered DOM after `data-ready` signals fire, and compute how often the output differs. That number is directly measurable and directly fixable. The ranking consequence is not: no external tool can observe a demotion field, and any claim that a specific consistency score predicts a demotion within a specific window — including the 30-to-60-day figure earlier versions of this article gave — is unfalsifiable. Track parity because divergence means the indexed document is not your page; do not promise yourself a timeline for the reward.
The warming run consumed roughly 90 hours of cumulative ostr.io render-pool time across 4 days, which billed at the platform's standard render-second rate (publicly listed). Just-in-time rendering for the same coverage would have cost approximately 60 percent more over the same window because peak Googlebot crawl bursts triggered cold-cache misses that re-rendered the same listing two or three times before settling. Batched warming amortizes the render across a controlled queue and avoids the duplicate-render tax.
The batch is throttled to respect the marketplace's origin server and the listing data API. ostr.io's batch warming queue paces requests so origin sees a sustained but bounded load (target: under 5 percent of normal user traffic). For 3.2M listings at the configured rate, 4 days is the natural completion window. Faster warming is available for sites that can absorb higher origin load, but for a live marketplace with real user traffic, 4 days was chosen as the safe operating envelope.
Not with a short TTL — ostr.io's TTL is per host with a 2-hour floor, so sub-hour clocks are not available. Freshness comes from webhook-driven invalidation on status changes, with the host TTL acting only as a backstop for changes no webhook covers. Webhook hooks (status: sold, status: archived, price-change events) fire immediate invalidations on the affected URL set, so meaningful changes propagate in under 60 seconds. The TTL only matters for changes that do not fire a webhook, which on a property marketplace is essentially metadata edits and image updates — neither of which are time-critical for indexation. ## Related Reading - DOM Consistency Check - Crawl Budget ROI: How Prerendering Pays for Itself - WAF Blocking Legitimate Bots: Cloudflare and AWS - Cache Warming API and Freshness Signal !Raster matrix diagram of operational levers, risks, and validation checks for Case Study: How a Marketplace Eliminated navDemotion Risk Across 3.2M Listings.
Explore the canonical guides
Editorial trust
Written by prerender Editorial · Engineering Team. We build and run pre-rendering infrastructure for more than 200 engineering teams, which is where the numbers and code samples on this page come from.
Last updated . Editorial scope and review policy: About prerender.info.