Crawl frequency signals - what makes crawlers come back

Introduction

1. 1

   Classify freshness-critical URLs before you touch signals

   Separate templates by business volatility first. A sensible baseline is: hourly-sensitive inventory pages such as job listings, booking availability, and status-changing real-estate listings; daily-changing pages such as products, categories, and changelogs; weekly-changing pages such as docs and comparison pages; and slow-moving pages such as about, legal, and static support content. Recrawl tuning only works when the freshness hierarchy is explicit.

2. 2

   Emit accurate lastmod only when page meaning changed

   Update sitemap `lastmod` when the page meaning changed, not on every deploy. Price, stock, `JobPosting.validThrough`, listing status, schema fields, primary copy, or visible inventory count are meaningful changes. A CSS tweak or analytics script update is not. If `lastmod` moves every time CI runs, crawlers learn to ignore it.

   textCopy

   1Update lastmod when one of these changes:
   2- visible body copy or title
   3- structured-data fields that affect eligibility
   4- price / stock / status / availability
   5- canonical target or major internal-link block

3. 3

   Make recrawls cheap with stable headers and stable HTML

   Recrawl frequency rises when the crawler learns that checking the page is cheap. Pair accurate `Last-Modified` or `ETag` with stable bot-facing HTML so revalidation can return `304 Not Modified` for unchanged pages. This is where pre-render cache headers matters most: freshness and cheap revalidation are two halves of the same system.

4. 4

   Increase internal prominence on pages where freshness matters

   A freshness-critical page hidden five clicks deep still teaches the crawler that it is low priority. Link active jobs from employer pages, active listings from city pages, and important docs from hub pages. Home, top nav, high-authority category pages, and sitemap shards all reinforce the same lesson: these URLs deserve revisits.

5. 5

   Return the right response code the moment the state changes

   Nothing destroys recrawl trust faster than stale or misleading removal signals. Use `410 Gone` for permanently removed URLs that should disappear fast, `301` for durable moves, real `404` for absent pages, and avoid soft 404s that return `200` with an empty shell. This is especially important on job boards and real estate, where validity and expiry are part of the page meaning.

6. 6

   Measure recrawl lag in 7, 30, and 90 day windows

   In the first 7 days, look for faster discovery and more conditional requests. In 30 days, watch whether freshness-critical templates are revisited sooner after publish or status changes. In 90 days, the right pattern is lower lag between content change and crawler refresh, fewer stale states left in the index, and a clearer separation between high-priority and low-priority URL families. If nothing changes, the usual reasons are bad lastmod hygiene, weak internal prominence, or soft-404 noise rather than a lack of crawler capacity.

Real recrawl intervals differ by template type

What teams mistake for a crawl-frequency problem

Freshness hierarchy should be explicit, not implied

• Guide - crawl budget fundamentals

  Budget and frequency work together, not separately.

• Guide - cache headers

  Cheap revalidation is one of the strongest recrawl signals you control.

• Guide - expand crawl budget

  Use this when frequency is only one part of the bottleneck.

• Use case - job boards

  Expiry and `validThrough` accuracy make recrawl urgency obvious.

• Use case - real estate

  Sold and withdrawn listings are the classic freshness test.

• Compare - vs Hadoseo

  Useful when freshness support is part of the vendor decision.