Key Takeaways
Google says moving to a CDN can raise Googlebot's crawl rate because faster responses lift its crawl capacity limit
A new CDN starts with a cold cache, so first requests for each URL still hit your origin server
Blanket bot protection on a CDN can block Googlebot and drop pages from Google Search and Shopping
Google's 3 fixes: allow verified bots, pre-warm caches, and return 429 or 503 for a short window
Google caps that 429 or 503 window at a few hours, or 1 to 2 days at most
Moving to a CDN can briefly flood your origin server with Googlebot traffic.
Google explained why in a Search Central blog post on 11 November, and listed 3 fixes.
The trigger is speed. Your CDN's edge answers faster than your old server, and Google reads that as room to crawl more.
AI agents add their own load to the same servers.
Microsoft's Clarity report on what AI agents do on your site helps you separate them from search crawlers.
Why does Googlebot crawl more after you move to a CDN?
Googlebot crawls more after a CDN move because Google's crawling systems set a crawl capacity limit, also called hostload, from how fast and stable your server responds. When your DNS points to a fast CDN edge, response times drop, and Google gradually raises the limit if there's crawl demand. Google's post compares the effect to your server getting a hardware upgrade.
Why your origin takes the hit
A new CDN starts with an empty, or cold, cache.
The first request for any URL at an edge location misses the cache, so the edge fetches it from your origin.
Visitors cluster on popular pages, which warm up fast. Crawlers work through your whole URL inventory, so many uncached URLs still land on the origin.
Google warns against answering the spike with blanket bot protection. Crawlers that hit CAPTCHA or challenge pages fail and back off.
If that goes on, pages can drop out of Search, according to Google. It names Shopping listings as affected too.
3 fixes Google lists for moving to a CDN
- Allow verified bots in your CDN and firewall settings, using Google's published crawler IP ranges.
- Use robots.txt or targeted rules to limit crawlers, and skip blanket blocking.
- Pre-warm caches before cutover: high-traffic URLs, upcoming peak pages and frequently crawled URLs.
- Fetch those URLs at a controlled pace, so warming doesn't become its own spike.
- If the origin still strains, return 429 or 503 through rate-limiting rules.
- Keep that 429 or 503 window to a few hours, or 1 to 2 days at most.
Google's guide to reducing Googlebot's crawl rate covers the status codes, and its Google request verification guide shows how to confirm a real Googlebot.
What Google said in 2024 and what it added in 2026
Google covered CDNs once before, in a December 2024 Crawling December post.
| Topic | December 2024 post | November 2026 post |
|---|---|---|
| Crawl rate on CDNs | Infrastructure allows higher crawl rates on CDN-backed sites | Explains the mechanism: faster responses raise hostload |
| Cold cache | Origin must serve each URL once; crawl rate likely high for a few days | Crawlers reach the long tail of URLs that visitors don't warm |
| Blocking | WAF blocks and bot interstitials can remove pages from the index | Blanket bot protection can drop pages from Search and Shopping |
| Temporary brake | 503 or 429 is the preferred temporary block | 429 or 503 for a few hours, or 1 to 2 days at most |
| Before cutover | Take care launching many URLs at once | Pre-warm high-traffic, peak and frequently crawled URLs |
The 2026 post adds 2 things you can act on: a time cap on the brake, and a pre-warm list.
What isn't confirmed yet
- How big a typical spike is, or how long it lasts. Google gives no figures.
- How long a block can run before pages drop. Google doesn't say.
3 things to do before your CDN cutover
Write the allowlist first. Add Google's verified crawler ranges to your CDN and firewall before you turn on any bot rules.
Warm the long tail. Pull your most-crawled URLs from Search Console's Crawl Stats report and pre-fetch them slowly.
Watch indexing for 2 weeks. Pages can fall out quietly. We've covered Google deindexing pages at higher rates, so check the Page indexing report daily after cutover.
Disclaimer: This content is generated with the help of AI and involves human oversight. Please verify the sources and claims at your end.




