Is Claude or ChatGPT Crawling Your Site Right Now?

Here's something that might surprise you: your analytics are probably lying to you.

Not maliciously. Google Analytics just wasn't built to show you bots. It filters them out. So while your dashboard shows a completely normal-looking traffic pattern, there's a whole other category of visitors you're not seeing at all. AI crawlers from OpenAI, Anthropic, Perplexity, Google, Meta, and others are almost certainly hitting your site on a regular basis, reading your content, indexing your pages, and in some cases using what they find to answer user questions in real time.

As of 2026, Meta-ExternalAgent is the second-most active crawler on the entire web, right behind Googlebot. GPTBot from OpenAI is third. ClaudeBot reportedly doubled its crawl rate between mid-2025 and early 2026. These are not occasional visitors. They're hitting sites across the web billions of times per day.

The question isn't whether they're visiting you. It's whether you know about it, and what you want to do.

Jump to Section

• Why Your Analytics Won't Show You This
• The AI Crawlers Worth Knowing
• How to Find Them in Your Server Logs
• Verifying a Crawler Is Legit
• Using Cloudflare and Vercel to See Everything
• What to Do Once You Know
• FAQ

---

Why Your Analytics Won't Show You This {#why-analytics-wont-show-you}

Google Analytics, GA4, Adobe Analytics, Matomo, all of them operate by loading a JavaScript snippet in the user's browser. Bots don't run JavaScript. They make direct HTTP requests to your server, read the HTML, and leave. The analytics tag never fires. The visit never gets counted.

This means everything you think you know about your traffic is only showing you the human portion. The bot portion, which on many sites runs between 30% and 40% of total server requests according to server log analysis studies, is completely invisible in your dashboard.

The only place this traffic actually shows up is your raw server logs or your CDN's analytics if you're running one. If you've never looked at those, you're operating without a significant chunk of the picture.

---

The AI Crawlers Worth Knowing {#ai-crawlers-worth-knowing}

Not all AI bots are doing the same thing. There are two main categories: training crawlers that collect data to build future AI models, and retrieval crawlers that fetch live content to answer user queries right now. The distinction matters because blocking them has different consequences.

The major ones you'll see in your logs:

GPTBot is OpenAI's training crawler. If it has visited your site, your content may have been used in training ChatGPT. User agent string: GPTBot/1.1. It generally respects robots.txt.

OAI-SearchBot is OpenAI's live retrieval crawler. This is what fetches your page in real time when a ChatGPT user asks something and the model needs current information. Separate from GPTBot, different purpose entirely.

ChatGPT-User is triggered by actual user conversations. When someone asks ChatGPT to read a URL or look something up, this agent makes the request. It can show up in your logs as bursty, irregular traffic.

ClaudeBot is Anthropic's primary training crawler. Full user agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). Anthropic also runs Claude-User for user-triggered fetches and Claude-SearchBot for search indexing. Each has a distinct job and can be controlled independently in your robots.txt.

PerplexityBot is what Perplexity uses to index and retrieve content for its answers. Since Perplexity is heavily citation-based in its responses, this crawler directly determines whether your content shows up as a source.

Google-Extended is Google's AI training crawler, separate from standard Googlebot. Blocking this opts you out of AI Overviews and Gemini training without affecting your regular search rankings.

Meta-ExternalAgent is Meta's crawler for AI training. It's currently the second-highest-volume crawler on the web. Most people have no idea it's visiting their site constantly.

Bytespider is ByteDance's crawler, feeding TikTok's AI products. It has a reputation for being less respectful of robots.txt than the US-based bots.

---

How to Find Them in Your Server Logs {#find-them-in-server-logs}

Your server logs are in different places depending on your setup. On Apache it's typically /var/log/apache2/access.log. On Nginx it's usually /var/log/nginx/access.log. If you're on a shared host, you can usually find them in your control panel under a "Logs" section.

Each line in an access log looks something like this:

66.249.66.1 - - [20/May/2026:14:32:11 +0000] "GET /blog/some-post HTTP/1.1" 200 4521 "-" "GPTBot/1.1"

That last field is the user agent string. That's what you're hunting for.

To pull all AI crawler activity out of your Apache logs in one shot, you can run:

grep -iE "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Meta-ExternalAgent|Bytespider|OAI-SearchBot|ChatGPT-User" /var/log/apache2/access.log

To get a count of how many requests each crawler made, sorted by volume:

grep -iE "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Meta-ExternalAgent" /var/log/apache2/access.log \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn

If you're on Nginx, same commands, just change the log path.

To see which specific pages each crawler hit most, filter by bot and pull the URL field:

grep "GPTBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

This tells you exactly which content OpenAI has been most interested in. That's actually pretty useful information if you're trying to optimize for AI citation.

If you're running Vercel like many Next.js sites do, raw access logs aren't exposed by default. You'll need either Vercel's Log Drains feature (available on Pro and Enterprise plans) to pipe logs somewhere queryable, or rely on a CDN layer in front of it for bot visibility.

---

Verifying a Crawler Is Legit {#verifying-a-crawler-is-legit}

Here's something important: user agent strings can be spoofed. Anyone can make a request that claims to be GPTBot. It means nothing on its own.

The way to verify a crawler is actually who it claims to be is through reverse DNS lookup. Take the IP address from the log entry and run:

host 23.102.140.113

For a legitimate GPTBot hit, that should resolve to a hostname in OpenAI's domain space. Then do a forward lookup on that hostname to confirm it resolves back to the same IP. If both directions check out, it's real. If the reverse lookup doesn't resolve cleanly to the company's domain, treat it as suspicious.

# Reverse lookup
host [IP_ADDRESS]

# Forward lookup to confirm
host [RETURNED_HOSTNAME]

OpenAI, Anthropic, Google, and Perplexity all publish their official IP ranges in their crawler documentation. Matching the IP against those published ranges is the most reliable method when you need to be certain.

---

Using Cloudflare and Vercel to See Everything {#using-cloudflare-vercel}

If digging through raw log files sounds tedious, a CDN layer makes this whole process significantly easier.

Cloudflare handles roughly 20% of all web traffic globally and its Bot Management product classifies AI crawlers automatically. Even the free Cloudflare plan gives you basic bot traffic visibility in the analytics dashboard. You'll be able to see how many requests are coming from known bots versus humans without touching a log file. The paid Bot Management tiers give you more granular detail and the ability to set rules per crawler.

One genuinely interesting feature Cloudflare is rolling out is pay-per-crawl, where you can charge specific AI crawlers a fee per request. It's early and niche right now but it's a concrete sign that the infrastructure layer is starting to treat AI bot traffic as something with real economic value rather than just overhead.

Vercel has its own bot detection called BotID that runs at the application layer. It identifies AI scrapers and automation without adding friction for real users. It doesn't expose full access logs by default but it will flag suspicious bot patterns on your routes.

For the most visibility with the least effort, the combination of Cloudflare in front plus your server logs as a fallback covers the vast majority of cases.

---

What to Do Once You Know {#what-to-do-once-you-know}

Finding out which AI crawlers are hitting your site basically opens three decisions.

The first is whether you want to be cited in AI answers. If yes, you want the retrieval and search crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot to have full access. Blocking those removes you from AI-generated responses. With 60% of Google searches now ending without a click, being cited in AI answers is increasingly how people discover sites at all. According to one study, 50% of AI citations come from content less than 13 weeks old, so freshness matters a lot here.

The second decision is whether you're okay with your content being used for AI model training. Training crawlers like GPTBot and ClaudeBot collect data to train future models. If you want to opt out of that without affecting your AI search visibility, you can block the training crawlers specifically while allowing the retrieval ones. Your robots.txt would look something like this:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Allow retrieval crawlers for AI citations
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

The third decision is about bandwidth and server load. Some crawlers, particularly Bytespider and CCBot, are known to be aggressive. If you're seeing thousands of requests from a bot that isn't offering any obvious benefit, rate limiting or blocking it at the server or firewall level is reasonable.

One thing worth understanding: you can use the HTTP headers tool to check what your site is actually returning to crawlers and make sure your server directives are being communicated correctly. If your robots.txt has errors or your server is returning unexpected headers to bots, your rules may not be applied the way you think.

The broader context here connects to what's happening across the web with AI search and ad revenue. If you're thinking about how AI is reshaping who gets traffic and who doesn't, check out the post on how AI is killing ad revenue. Understanding which crawlers are hitting your site is step one toward having any intentional strategy in that landscape. And if you're working on structured data or API responses to make your content more crawlable, the JSON formatter and API directory are worth bookmarking.

---

FAQ {#faq}

How do I know if GPTBot has crawled my site? Check your server access logs for the user agent string GPTBot/1.1. You can use a grep command to search: grep "GPTBot" /var/log/nginx/access.log. If you're on Cloudflare, the bot analytics dashboard will show this without requiring log access.

What is ClaudeBot and what does it do? ClaudeBot is Anthropic's primary web crawler. It collects content from public websites to train future versions of the Claude AI model. Its full user agent string contains ClaudeBot/1.0 and an email address. Anthropic also runs Claude-User (triggered by real user requests) and Claude-SearchBot (for search indexing), and they're all controlled independently in robots.txt.

Will Google Analytics show me AI crawler traffic? No. Google Analytics, GA4, and most tag-based analytics tools filter out bot traffic because bots don't execute JavaScript. AI crawlers make direct HTTP requests that never trigger your analytics tag. The only way to see them is through server logs, CDN analytics, or a WAF.

Can I block AI crawlers from scraping my site? Yes, mostly. Reputable crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended all respect robots.txt directives. Less reputable crawlers, particularly some offshore bots, may not. For reliable blocking you should combine robots.txt with server-level or firewall rules targeting the known IP ranges each company publishes.

Is it worth blocking AI crawlers? Depends on your goal. Blocking training crawlers keeps your content out of future model training, which is a reasonable data rights decision. Blocking retrieval crawlers removes your content from AI search citations, which in 2026 increasingly means disappearing from how people find information. Most site owners are better off blocking training crawlers they object to while allowing retrieval crawlers that drive visibility.

What is the difference between GPTBot and OAI-SearchBot? GPTBot is OpenAI's training crawler, collecting data to build future models. OAI-SearchBot is a retrieval crawler that fetches live content to answer real-time user queries. ChatGPT-User is a third agent triggered by actual user conversations. Blocking all three has very different consequences and they should be configured separately in robots.txt.

How do I verify a crawler is actually from the company it claims to be? Run a reverse DNS lookup on the IP address in your log. It should resolve to a hostname in the company's domain. Then do a forward lookup on that hostname to confirm it points back to the same IP. User agent strings alone can be spoofed; IP verification is the reliable check.

Can AI crawlers affect my server performance? Yes. Some crawlers, particularly Meta-ExternalAgent, Bytespider, and CCBot, can be aggressive in their crawl rates. If you're on limited hosting or running a high-traffic period, aggressive AI crawlers can add meaningful load. Monitor your total request volume in server logs or CDN analytics and rate limit or block high-volume bots if they're causing problems.

What pages do AI crawlers prioritize? Based on observed behavior, AI crawlers tend to prioritize recently updated content, pages with clear structured data, content with high external link authority, and pages that answer specific questions well. According to one study, 50% of AI citations come from content less than 13 weeks old, which means freshness is a significant factor in whether you get cited at all.

If I run a Next.js site on Vercel, can I see AI crawler traffic? Not easily by default. Vercel doesn't expose raw access logs unless you set up Log Drains. The easiest solution is to put Cloudflare in front of your Vercel deployment, which gives you bot traffic visibility on Cloudflare's free plan. Vercel's BotID feature can also detect and flag suspicious crawler activity at the application layer.

Does robots.txt actually stop AI crawlers? The major US-based crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) generally respect robots.txt. Smaller crawlers and offshore bots are less reliable about it. TollBit has reported that billions of AI bot scrapes have bypassed or ignored robots.txt entirely. For content you seriously need to protect, robots.txt alone isn't enough and you need server-side controls.

Should I update my robots.txt to include AI crawler rules? Yes, especially if you have content you'd prefer not to be used for training. Even if you're fine with AI crawlers in general, being intentional about which ones have access to what is better than running on unexamined defaults. At minimum, add explicit rules for GPTBot, ClaudeBot, Google-Extended, and PerplexityBot so your intent is clear.