AI Crawler Traffic in Google Analytics: Finding the Ghost Visits

If you have gone looking for AI crawler traffic in Google Analytics, you have probably found very little of it, and that is expected. Most AI crawlers never run the JavaScript that GA4 depends on, so the automated sessions you can see are something else: headless browsers, agents and scrapers running a real browser from cloud networks. This guide explains the difference, how to spot both kinds, and what changed at Cloudflare on 15 September 2026.

What is a ghost visit?

A ghost visit is a session in your analytics or visitor feed that was generated by software rather than a person. It might be a scraper copying your pricing page, an AI agent reading a product page on someone's behalf, an uptime monitor, an SEO audit tool or a headless browser running a test script.

Ghost visits matter for two reasons. They inflate traffic and deflate conversion rates, which makes good pages look bad. And if you use a visitor-identification tool, they can surface as "companies" that were never prospects: a cloud provider, a hosting firm or a monitoring service.

How much web traffic is automated in 2026?

More than half of it, by one large measure. On 3 June 2026, Cloudflare's data showed automated requests at 57.5% of HTTP requests to web content, against 42.5% from humans, which Tech Times described as the first time since the web opened to the public that machines generated most requests¹. Cloudflare's chief executive, Matthew Prince, reacted with "Welp, that happened faster than I predicted"¹.

NBC News, reporting on 4 June 2026, put the split at 57.4% bots to 42.6% humans across requests to websites Cloudflare serves, and quoted Prince noting that the classification is "a bit messy" and that "bots don't click on ads"². The exact decimal varies by day and source. The direction does not.

That headline figure is about requests hitting servers. It is not a measure of what reaches your analytics, which is where the story gets more useful.

Do AI crawlers show up in Google Analytics?

Mostly not. GA4 records a visit when its tag, a piece of JavaScript, runs in the visitor's browser. A crawler that downloads your HTML but never executes the scripts in it never fires the tag.

The clearest evidence comes from a study Vercel published with MERJ on 17 December 2024. Across Vercel's network in the month studied, GPTBot made 569 million requests and Anthropic's crawler 370 million³. The study found that "none of the major AI crawlers currently render JavaScript", covering bots from OpenAI, Anthropic, Meta, ByteDance and Perplexity. ChatGPT's and Claude's crawlers did fetch JavaScript files, but did not execute them³. The exceptions at the time of the study were Google's Gemini, which uses Googlebot's infrastructure, and AppleBot, which renders pages through a browser-based crawler³.

GA4 also filters traffic from known bots automatically. Google says this uses "a combination of Google research and the International Spiders and Bots List" from the IAB, that you cannot switch it off, and that you cannot see how much was excluded⁴.

So the answer to "is AI bot traffic showing in analytics?" is usually no. The crawlers show up in your server logs and your CDN dashboard, not in GA4. The same logic applies to any visitor-identification snippet that runs as JavaScript: a crawler that does not execute scripts never triggers it.

Then what are the ghost visits?

The automated sessions that do reach GA4 come from software that runs a real browser engine. Chrome's own documentation describes headless mode as a way to "run the browser in an unattended environment, without any visible UI", and it is the basis of automation and testing tools such as Puppeteer and Selenium⁵. A headless browser loads the page, runs every script and fires every tag, just like a person's browser would.

That covers a wide mix of software:

• AI agents that browse on a user's behalf, opening pages in a real browser to read or act on them.
• Scrapers and price monitors built on headless Chrome.
• SEO and accessibility audit tools, uptime checks and synthetic performance tests.
• Your own team's test scripts, if they run against the live site.

Almost all of these run from cloud and data-centre networks rather than from a company office or a home broadband line. That is the most reliable tell.

How to spot them in GA4 and server logs

No single signal is conclusive, so look for several together.

The trap is the user-agent string. It is free text that any client can set, and many scrapers claim to be an ordinary browser, or even Googlebot. Google warns that "spammers or other troublemakers" access sites "while claiming to be from Google", and recommends verifying Googlebot with a reverse and forward DNS lookup or by matching the IP against its published ranges⁶. OpenAI likewise publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User⁷.

The practical rule: count bots by the network they come from, not by the name they give. If a session says it is Chrome on Windows but comes from a hosting provider's IP range, treat it as automated.

What changed on 15 September 2026?

On 1 July 2026 Cloudflare announced new controls that split AI traffic into three categories, Search, Agent and Training, and set new defaults from 15 September 2026⁸⁹. In Cloudflare's own words: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."⁹ TechCrunch reported that the change to block "mixed-use" crawlers on ad-bearing pages also covers new sites from existing customers and all existing free-tier customers⁸.

The mixed-use part matters most. Cloudflare says that "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training"⁹. If you run a lead-generation site on Cloudflare and have switched on training blocks, check that you have not also blocked the crawlers that put you in search results.

This builds on Cloudflare's Content Signals Policy, published on 24 September 2025, which adds lines to robots.txt separating three uses: search, ai-input (using content in AI answers) and ai-train¹⁰. An example line is Content-Signal: search=yes, ai-train=no. Cloudflare is clear that "content signals express preferences; they are not technical countermeasures against scraping"¹⁰.

For a B2B site that wants to be found and cited, a sensible starting position is:

• Allow search-role bots. OpenAI says OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features", and sites that opt out will not appear in those answers⁷.
• Decide separately about training. GPTBot is the crawler OpenAI uses for content "that may be used in training our generative AI foundation models"⁷. Blocking it is a business choice, not a traffic fix.
• Know that user-triggered fetchers behave differently. OpenAI notes that for ChatGPT-User, "robots.txt rules may not apply" because a person initiated the action⁷.

Why the crawl-to-refer ratio matters

Cloudflare's crawl-to-refer ratio compares how often an AI platform crawls a site with how often it sends visitors back¹¹. For the week of 19 to 26 June 2025 it put Anthropic at about 70,900 crawls per referral, while noting the figure may be overstated because native app traffic often carries no referrer¹¹. Ratios move a lot between windows, so treat any single number as a snapshot. The useful takeaway is that crawling costs you bandwidth, and only some crawlers return traffic.

How should a visitor-identification tool treat bots?

It should score automated traffic at zero and leave it out of the count. A cloud provider appearing as a "hot lead" is not a lead: it is a server somewhere running a script.

That is how Scout works. Bots are excluded, scored at zero and not counted as a company. ISP, home broadband, mobile, VPN and data-centre matches are flagged and can never score above the low band of the 1 to 99 confidence score. Your plan is measured in identified visits, and a visit only counts when Scout names the company behind it; see how plans are measured.

It also helps to know where the other noise comes from. Home broadband providers are the other big source of misleading matches, covered in why ISPs show up as website visitors. For the people who are real but arrive from AI answers without a referrer, see how AI search changes B2B clicks. And for the bigger picture, read our guide to website visitor identification.

Seeing past the ghosts

AI crawlers mostly stay out of GA4, while headless browsers and agents slip in from cloud networks. Check your logs by network rather than by name, keep search-role crawlers allowed if you want to be cited, and use tools that score automated traffic out of your reports.

Want to see which of your visits are people, and which are ghosts? Start a 14-day free trial, no card needed.

Frequently asked questions

Should I block AI crawlers?

Block by role, not wholesale. If you want to be cited in AI search answers, keep search-role crawlers such as OAI-SearchBot allowed. Blocking training crawlers such as GPTBot is a separate business decision about how your content is used.

Does bot traffic count towards my Scout identified-visits plan?

No. Scout scores automated traffic at zero and does not count it as a company, and a visit only counts towards your plan when Scout names the company behind it. Visits Scout cannot identify are free.

Why does my crawl-to-refer ratio matter?

It shows how much an AI platform takes from your site compared with the visitors it sends back. A very high ratio means you pay the bandwidth cost of crawling with little referral traffic in return. Treat any figure as a snapshot, because ratios change between measurement windows.

Why can't I see GPTBot in GA4?

GA4 only records a visit when its JavaScript tag runs, and the major AI crawlers did not execute JavaScript at the time of the Vercel and MERJ study. GA4 also removes known bots automatically. Look in your server or CDN logs instead.

Sources

1. Tech Times: Bot Traffic Passes Humans Online: Cloudflare Says Agentic AI Drove 57.5% Share techtimes.com
2. NBC News: Bot web traffic has overtaken human web traffic, data shows nbcnews.com
3. Vercel: The rise of the AI crawler vercel.com
4. Google Analytics Help: Known bot-traffic exclusion support.google.com
5. Chrome for Developers: Chrome Headless mode developer.chrome.com
6. Google Search Central: Verifying Googlebot and other Google crawlers developers.google.com
7. OpenAI: Overview of OpenAI crawlers developers.openai.com
8. TechCrunch: Cloudflare's new policy pushes AI companies to pay for publishers' content techcrunch.com
9. Cloudflare blog: Your site, your rules blog.cloudflare.com
10. Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy blog.cloudflare.com
11. Cloudflare blog: The crawl-to-refer ratio for AI search blog.cloudflare.com