How to move your site to a CDN without hurting SEO

Technical SEO · 20 November 2026 · 8 min read

Why Googlebot crawls harder after a CDN move, how bot protection can lock it out, and a checklist to pre-warm caches and keep Google crawling safely.

Server racks with network cables running between them

A CDN migration can hurt SEO in two ways. Google may crawl faster than your origin server can handle while the CDN's cache is still empty, and the bot protection you turn on to stop that traffic may block Googlebot itself. Google explained both, and how to avoid them, in a Search Central post in November 2026, written by Gary Illyes and Cherry Prommawin. This guide explains why they happen and what to do before, during and after the switch.

Why does Googlebot crawl more after a CDN move?

Google sets a crawl capacity limit for every site, also called hostload. According to Google's crawl budget documentation, every site starts with the same conservative limit. It goes up when responses stay fast and stable, and down when the site slows or returns 5xx errors or 429 responses.

When your DNS starts pointing at a CDN, response times drop sharply. To Google's crawlers it looks as if your server got a large hardware upgrade, so they raise the crawl rate step by step. That increase is usually welcome. The trouble is where the requests end up during the first days.

Why does the origin server still take the load?

A new CDN has an empty cache. The first request for any URL at an edge location is a cache miss: the edge server fetches the page from your origin, keeps a copy, then serves it. Only the second request for that URL is served from the edge alone.

Human visitors mostly request the same few pages (the homepage, popular articles, current offers), so those warm up within hours. Crawlers do the opposite. They work through your whole URL inventory, including old articles, filtered category pages and paginated archives that no person has opened in months. Every one of those first requests goes straight to the origin, now at a faster crawl rate than before.

Three details of CDN setup make this worse:

  • Each edge location has its own cache. A page cached in Mumbai is still a miss in Virginia. Google says its crawlers egress primarily from IP addresses in the United States, so for an Indian site, the edges Googlebot reaches are often not the ones your visitors warm up.
  • Many CDNs don't cache HTML by default. Cloudflare, for example, caches static files such as images, CSS and JavaScript by file extension and sends HTML to the origin unless you add a cache rule. If your pages are HTML, the CDN may be shielding very little.
  • Query strings create separate cache entries. If ?utm_source= or sort and filter parameters are part of the cache key, one page can become hundreds of cache misses.

How does bot protection end up blocking Google?

Right after a migration, an admin sees a wall of automated requests in the origin logs, assumes a botnet is scraping the site and switches on the CDN's strictest bot setting. Without exemptions, that setting treats every automated client the same. Googlebot gets a CAPTCHA or JavaScript challenge, or a 403 error, and it can't pass any of them.

Some challenge pages are served with a 200 status code. Google's crawlers can't tell that the page is a security check, so to them the site looks broken or deliberately hidden. If the block stays in place, Google crawls less and starts dropping pages from the index. In the video that accompanied the post, Google adds that products disappear from Google Shopping and ads products stop working. Google hasn't said how long a block has to last before that happens.

The same settings block AI search crawlers too. ChatGPT, Perplexity and Claude each fetch pages with their own crawlers, and a blanket bot rule shuts all of them out. If being cited in AI answers matters to you, check those crawlers as well. Crawl parity between search bots and LLM bots covers this in more detail.

What to set up before you switch

Decide what the CDN caches

Write down which URLs should be cached, and for how long, before any traffic moves:

  • Add a cache rule for HTML pages if your CDN doesn't cache them by default, and keep logged-in, cart, checkout and other personalised pages out of the cache.
  • Leave tracking parameters such as utm_* and gclid out of the cache key, so they don't create extra copies of the same page.
  • Set Cache-Control headers at the origin that match how often each page type changes, and check whether the CDN respects them or overrides them.

Turn on tiered caching or an origin shield

Most large CDNs can route cache misses through one intermediate layer instead of sending every edge location to your origin separately. Cloudflare calls it Tiered Cache, Fastly calls it shielding, and Amazon CloudFront calls it Origin Shield. With it switched on, a page fetched once from the origin can fill caches worldwide, which cuts origin load during the warm-up period.

Allow verified crawlers before you enable any bot rule

Most CDNs and web application firewalls have a setting or managed rule that lets verified bots through while still filtering everything else. Turn it on first, then decide what else to block. Google recommends a more targeted approach where you can: block specific unwanted bots in robots.txt or with firewall rules, and leave other automated clients alone.

Check two things on that allow list:

  • It should include Google's special-case crawlers as well as Googlebot. AdsBot, which checks Google Ads landing pages, comes from a separate IP range. Google publishes both lists on its crawler verification page as common-crawlers.json and special-crawlers.json.
  • Keep robots.txt and your XML sitemaps out of every challenge. If Google can't fetch robots.txt because of a server error, it stops crawling the site until it can.

Pre-warm the cache

Before routing all traffic through the CDN, fetch your most important URLs through it so the cache is full when Googlebot arrives. Google suggests three groups: your highest-traffic pages, pages about to get a traffic peak (a sale, a seasonal launch, a product release), and your most frequently crawled URLs.

Build the list from your XML sitemaps, the top pages in Search Console and the URLs Googlebot requests most often in your server logs. Put them in a file called urls.txt, one full URL per line, and request them through the CDN at a steady pace:

while read -r url; do
  curl -s -o /dev/null -w "%{http_code} %header{cf-cache-status} $url\n" "$url"
  sleep 0.5
done < urls.txt

Run it twice. The first pass should show MISS and the second HIT. Other CDNs use a different header, usually x-cache. To warm the edges Googlebot is likely to reach, run the script from a server in the United States as well as from India. With tiered caching on, one pass fills the upper tier for every edge location.

What to do if the origin still struggles

Even with a warm cache and pacing on the CDN's side, crawling can put real load on a small origin during the first week. Google's guide to reducing the crawl rate gives one emergency brake: return a 503 or 429 status code to crawl requests, with a Retry-After header saying when to come back.

HTTP/1.1 503 Service Unavailable
Retry-After: 120

Keep it short. Google says a few hours, or one to two days at most. When it sees a significant number of these errors, it slows crawling across the whole hostname, then speeds up again by itself once they stop. URLs that keep returning server errors for longer than that can drop out of the index.

Two things not to do: don't return 403 to slow Google down, because that tells it the content is forbidden; and don't return 404, because that tells it the page is gone. If serving errors isn't possible on your setup, Google has a form to report an unusually high crawl rate. It can take several days to process, and it only lets you ask for less crawling, never more.

How to check that Googlebot isn't blocked

In the first two weeks after the switch, run these checks every few days:

  1. URL Inspection in Search Console. Run a live test on a few important pages. It fetches from Google's own infrastructure, so if the screenshot or HTML shows a challenge page, the CDN is blocking Google.
  2. Crawl Stats report (Search Console, Settings, Crawl stats). Watch total crawl requests, average response time and the breakdown by response code. A rise in 403 or 5xx responses, or host status warnings, needs attention the same day.
  3. Server and CDN logs. Filter requests that claim to be Googlebot and confirm they're real with a reverse DNS lookup on the IP, then a forward lookup on the hostname it returns. Genuine crawlers resolve to googlebot.com, google.com or googleusercontent.com. If verified Google IPs are getting 403s or challenges, fix the rule. How to analyze server log files for SEO covers the method.
  4. Merchant Center and Google Ads, if you sell online. "Product page unavailable" errors or disapproved landing pages soon after the switch usually mean the CDN is blocking Google's shopping or ads crawlers.
  5. AI crawlers. Run your homepage through the AI visibility checker to see whether ChatGPT's, Perplexity's and Claude's crawlers can still reach it.

A short dip in crawling right after the switch is normal. Google's guide to moving a site without changing URLs says to expect a temporary drop, then a steady rise over the following days. A sharp fall that doesn't recover, or a rise in 403 responses, points to a blocking problem.

Mistakes that cause trouble

  • Turning on "block all bots" or "under attack" modes in a panic, with no verified-bot exemption
  • Allowing Googlebot but forgetting AdsBot and the other special-case crawlers
  • Assuming the CDN caches HTML when it only caches images, CSS and JavaScript
  • Pre-warming only from India when Google crawls mostly from the United States
  • Using 403 or 404 responses to slow crawling, or leaving 503s in place for a week
  • Moving to the CDN days before a big sale, with no time to watch the results

Key takeaways

  • Allow verified crawlers, including Google's special-case crawlers, before you turn on any bot protection.
  • Check whether your CDN caches HTML, and take tracking parameters out of the cache key.
  • Turn on tiered caching or an origin shield to cut origin load while caches fill.
  • Pre-warm your top, upcoming peak and most crawled URLs, including from a US location.
  • If the origin needs a break, use 503 or 429 with Retry-After for a few hours, never 403 or 404.
  • Run URL Inspection live tests and watch the Crawl Stats report for the first two weeks.

FAQ Questions

Questions people ask

Does moving to a CDN improve SEO?

Not directly. Google doesn't rank a site higher for using a CDN. Faster pages help visitors and page experience, and a fast, stable server lets Google crawl more of the site, which matters most for large sites that change often.

How long does it take to warm up a CDN cache?

It depends on how many URLs you have, how long each page stays cached and how many edge locations serve your traffic. Google hasn't published a figure. Pre-warming your important URLs before the switch shortens the period when the origin carries the load.

Should I block bots on Cloudflare?

You can block the ones you don't want, but turn on the verified bots exemption first so Googlebot, Bingbot and other legitimate crawlers get through. Blocking specific bots by name in robots.txt or with firewall rules is safer than a blanket block on all automated traffic.

Will a new IP address affect my rankings?

No. Moving to a CDN changes the IP addresses your domain resolves to, and Google treats it like a hosting change. As long as the URLs stay the same and Google can reach them, rankings aren't affected. Expect a short dip in crawling, then a rise over the following days.

What status code should I use to slow Googlebot down?

Use 503 or 429, ideally with a Retry-After header, for a few hours or one to two days at most. Google then slows crawling across the whole hostname and speeds up again once the errors stop.

Chaitanya Kore in round sunglasses, facing the camera

Author About the writer

Chaitanya Kore

AI & SEO Strategist · Pune · Since 2017

I have worked in SEO since 2017, starting as an analyst. I have led an SEO team of 19 analysts across more than 70 projects for clients in India and Sweden. Today I run SEO strategy, including AEO and GEO, for international SaaS, eCommerce and B2B clients at WSI ALM Corp.

I also work directly with businesses in Pune on a 12-month SEO, AEO and GEO contract, planned one quarter at a time. If anything on a quarter's list isn't delivered, the extra hours are on me.

BBA, Marketing, Pune University · Certified: Google Analytics, HubSpot

Next Get in touch

Plan your first quarter

Based in Pune and working with companies across India. If anything on a quarter's list isn't delivered, the extra hours are on me.