Question 1

Why does GA4 not show AI crawler traffic?

Accepted Answer

GA4 does not show AI crawler traffic because it filters known bots and spiders before the data is recorded, following the IAB Tech Lab Spiders and Bots list by default. The setting is enabled in every property unless explicitly disabled, and even when disabled the GA4 collection model relies on client-side JavaScript that most AI crawlers either do not execute or execute in a way that produces unreliable signals. GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and OAI-SearchBot all either skip JavaScript execution entirely or render in headless modes that GA4 cannot reliably distinguish from human visitors. The only durable source of truth is the raw server access log, where every request — bot or human — is recorded with user-agent, IP address, response code, and bytes served before any client-side filtering happens.

Question 2

What is the difference between ChatGPT-User and OAI-SearchBot?

Accepted Answer

ChatGPT-User is the user-agent OpenAI uses when a ChatGPT user explicitly triggers a browse action inside a conversation — it represents real-time on-demand fetches initiated by an end user. OAI-SearchBot is the crawler OpenAI uses to build and refresh the index that powers ChatGPT search results, similar in spirit to Googlebot for classical search. The distinction matters operationally because ChatGPT-User volume correlates with how often your site is referenced inside live ChatGPT sessions and is a leading indicator of citation surface, while OAI-SearchBot volume reflects index coverage and freshness. According to OpenAI's official documentation at platform.openai.com/docs/bots, both crawlers respect robots.txt directives but should be treated as separate signals when measuring AI search exposure. Conflating them in a single bucket loses the user-intent signal that ChatGPT-User uniquely provides.

Question 3

How do I detect user-agent spoofers pretending to be AI crawlers?

Accepted Answer

Detect user-agent spoofers by reverse-DNS verification, ASN matching, and signed IP range lists published by the crawler operators. A request claiming to be GPTBot is only legitimate if its source IP resolves back to an OpenAI-controlled hostname or sits inside the published OpenAI IP range. Google publishes verified IP ranges for Googlebot and Google-Extended at developers.google.com, OpenAI publishes ranges for GPTBot and OAI-SearchBot, and Cloudflare maintains a verified bots program at radar.cloudflare.com/verified-bots that aggregates verified ranges for over 200 known crawlers. Any request with an AI crawler user-agent that fails reverse-DNS lookup or sits outside the published range should be classified as a spoofer and excluded from your citation dashboards. In practice, roughly 8 to 14 percent of requests claiming to be GPTBot in mid-sized commercial sites are spoofed.

Question 4

What fields should I retain in server logs for AI crawler analysis?

Accepted Answer

Retain at minimum the following fields for every request: timestamp at millisecond precision, source IP address, user-agent string, request method and full path, response status code, bytes served, referrer, request processing time, and the autonomous system number derived from the source IP. The ASN is essential because user-agent strings can be spoofed but the network the request originates from cannot. Cloudflare HTTP logs and Fastly real-time logs expose ASN natively. For Nginx and Apache, derive ASN with a streaming enrichment step using a maintained MaxMind or IPinfo dataset. Retain ninety days of logs at minimum, ideally one year, because the lag between an AI crawler fetching a page and that page being cited in a synthesized answer can run anywhere from twenty-four hours to roughly eight weeks depending on the crawler and the assistant.

Question 5

How often should I refresh my AI crawler citation dashboard?

Accepted Answer

Refresh your AI crawler citation dashboard daily, ideally on a fixed morning schedule that aligns with your team's standup or daily review cadence. Daily refresh catches crawler behavior shifts within twenty-four hours, which is the fastest meaningful signal cycle given that most AI search indexes refresh on rolling daily or sub-daily cadences. Refresh more frequently than daily only if you operate a high-velocity news or commerce site where citation freshness directly drives revenue and a six-hour lag would materially shift decisions. For most operators, daily is enough to detect when a new crawler appears, when an existing crawler changes its fetch pattern, or when spoofing volume spikes. The companion piece on the [AI search competitive intelligence daily standup](/article/ai-search-competitive-intel-daily-standup-2026) describes the meeting cadence that consumes this dashboard.