DNS and edge topology for crawler routing

Introduction

1. 1

   Record what your zone answers today, not what the dashboard says

   Dashboards show intent. Resolvers show reality, and the two drift after every hurried change. Query the public answer for each hostname that serves indexable content, and record the TTL alongside the value — the TTL is what will bound every change you make later.

   zone-snapshot.sh

   bashCopy

   1for host in example.com www.example.com shop.example.com; do
   2  # value + TTL, straight from a public resolver
   3  dig @1.1.1.1 +noall +answer "$host" A | awk '{print $1, "TTL="$2, $4, $5}'
   4done
   5
   6# Who is authoritative, and is the zone signed?
   7dig +short NS example.com
   8dig +short DS example.com   # empty = DNSSEC not enabled at the registrar

2. 2

   Decide which hostnames resolve to the edge and which do not

   A proxied record answers with an edge address and keeps the origin address as a private property of that record. An unproxied record publishes your server. The choice is per hostname, and it is worth being explicit: mail, SSH, staging, and API hosts frequently should not be proxied, while every hostname serving indexable HTML almost always should. Cloudflare exposes this as the orange-cloud proxy state; Bridge DNS — operated by the same team as this site — exposes it as a per-record toggle with origin port, protocol, and `Host` header set next to the record.

3. 3

   Set TTLs to match the rollback window you actually want

   TTL is the only lever you have over how long a mistake survives. A record at 3600 seconds means resolvers may serve the old answer for an hour after you fix it, and no provider can purge public resolver caches. Publish a low TTL *before* a planned change, not during it: resolvers only learn the new TTL when their cached copy expires, so the short value has to be live for longer than the old TTL to take effect everywhere.

   ttl-plan.txt

   bashCopy

   1T-48h   Set TTL to 300 on every record the change will touch.
   2T-24h   Verify propagation of the *TTL itself*, not just the value:
   3          dig @8.8.8.8 +noall +answer www.example.com | awk '{print $2}'
   4          dig @1.1.1.1 +noall +answer www.example.com | awk '{print $2}'
   5        Both should print 300 or less before you proceed.
   6T-0     Make the change. Rollback window is now ~5 minutes, not ~1 hour.
   7T+24h   Once the new value is stable, restore the normal TTL.

4. 4

   Verify the crawlers you allow, using both halves of the reverse check

   Allowlisting a User-Agent string allows anyone who can type it. Forward-confirmed reverse DNS is the check that actually holds: resolve the request IP to a `PTR` name, confirm the name belongs to a domain the crawler publishes, then resolve that name back and confirm it returns the original IP. Cache the verdict per IP — the check costs two round trips, and re-running it on every request puts a DNS lookup in your hot path.

   verify-crawler.sh

   bashCopy

   1IP="66.249.66.1"
   2
   3NAME=$(dig +short -x "$IP" | sed 's/\.$//')
   4case "$NAME" in
   5  *.googlebot.com|*.google.com|*.search.msn.com) ;;
   6  *) echo "reject: PTR '$NAME' not a published crawler domain"; exit 1 ;;
   7esac
   8
   9# Forward-confirm: this is the half that makes the check meaningful
   10dig +short "$NAME" | grep -qx "$IP" \
   11  && echo "verified: $IP" \
   12  || echo "reject: forward lookup does not match $IP"

5. 5

   Confirm the origin refuses traffic that skipped the edge

   Proxying the record hides the origin address from public answers; it does not stop anyone who already knows it. Historical DNS archives, TLS certificate transparency logs, old mail headers, and a misconfigured subdomain all leak origin addresses routinely. Close the loop at the origin firewall: accept inbound 80/443 only from your edge provider's published ranges, and reject everything else.

   origin-exposure.sh

   bashCopy

   1# 1. Public answer must be an edge address
   2dig +short www.example.com
   3
   4# 2. From a host outside your network, force the origin IP with the real Host
   5curl -sS -o /dev/null -w '%{http_code}\n' \
   6  --resolve www.example.com:443:198.51.100.25 \
   7  https://www.example.com/
   8
   9# 200 here means the WAF, the bot rules, and the rate limits
   10# above it are all optional from an attacker's point of view.
   11# Expect a connection timeout or a 403 from the origin firewall.

6. 6

   Write the rollback path down before you need it

   Every rollout has a slowest reversible layer, and that layer defines your incident response time. A proxy upstream or a route flag reverts in seconds. A record change reverts in one TTL. A nameserver change reverts on the registrar's and the TLD's schedule, which is not yours. Decide which layer your rollout touches, write the revert command in the runbook, and if the answer is 'nameservers', schedule the change for a week you can afford to watch.

An exposed origin makes the layers above it optional

What to ask a DNS provider when the zone carries a crawler path

What this layer cannot do for you

• WAF over-blocking fix

  The firewall layer directly above this one, provider by provider.

• Pre-launch checklist

  Where these checks fit in a go-live sequence.

• Bot detection and offloading

  Classifying crawler traffic once it reaches the edge.

• Guide — cache headers

  Cache-Control and Vary once routing is settled.

• Guide — large sites (100k+ pages)

  Where hostname and sharding decisions start to matter.

• Compare — vs Cloudflare

  Build-vs-buy at the edge layer these records point to.