AI Crawlers and Bot Traffic 2026: GPTBot, ClaudeBot, Crawl-to-Refer Ratios and robots.txt

For twenty years the deal was simple: search engines crawl your website and send visitors back. AI changed the terms. Crawlers from OpenAI, Anthropic, Meta and others now read the web on a massive scale to train models and to answer questions, but they return only a fraction of the traffic. We analysed the latest Cloudflare Radar data (four weeks to 27 October 2026) and the official crawler documentation of OpenAI, Anthropic, Google and Mistral: who crawls how much, for what purpose, how many visitors come back and what you should put in your robots.txt.

The web now has a second audience

In the four weeks to 27 October 2026, Cloudflare classified only 34.9% of HTML page requests as human. Classic bots (search engines, monitoring, scrapers) made up 54.7%, AI bots 6.4% and mixed-purpose crawlers 4%. Cloudflare reported in July 2026 that more than half of Internet traffic is now non-human, and Imperva's 2026 Bad Bot Report puts automated traffic at more than 53% of all web traffic in 2025, up from 51% a year earlier.

Among AI-related crawlers, Googlebot remains the largest single bot at 23.2% of the traffic, followed by ClaudeBot (12.6%), Meta-ExternalAgent (11.4%), Applebot (8.1%) and Bingbot (7.3%). In July 2025 GPTBot and ClaudeBot together accounted for about half of AI-only crawling, according to Cloudflare.

Crawl-to-refer: how much comes back

Cloudflare's crawl-to-refer ratio compares the HTML pages a platform's bots request with the HTML page visits the same platform refers to websites. The lower the number, the fairer the exchange. Classic search engines are close to balance: Google crawls 4.7 pages per visitor it sends, DuckDuckGo 4.1. AI platforms are two to three orders of magnitude higher.

Changes against the previous four weeks were large: OpenAI's ratio rose 145.8%, Perplexity's 72.7%, while Anthropic's fell 12.7%. Short windows move a lot, so read the level, not the weekly swing.

Anthropic: from 287,000 to 487 crawls per visitor

The long-term trend is clearer. In January 2025 Anthropic crawled almost 287,000 pages per referred visit, by far the most imbalanced ratio of any major platform. Web search in Claude (from March 2025, for all users from May 2025) added clickable citations, and the ratio fell to 38,000 by July 2025. In October 2026 it stands at 487:1. OpenAI improved only slightly over the same period (1,217 to 882), while Perplexity went the other way, from 55 to about 4,000.

What the crawlers are for: training dominates

Cloudflare classifies AI crawlers by purpose using operator disclosures. In the last four weeks 44.2% of AI crawler traffic was for training, 38.7% came from mixed-use crawlers (training and search in one bot, such as Googlebot, Applebot and Bingbot), 11.6% was pure search, 4.1% user actions (a chatbot fetching a page because someone asked) and 1.5% undeclared. Cloudflare's July 2026 report put training at 52% of crawler requests in June 2026, up from 22% in spring 2025.

The split inside each provider shows why the crawl-to-refer ratios differ. Of Anthropic's crawl requests, 84.5% come from ClaudeBot (training), 15.2% from Claude-SearchBot and 0.3% from Claude-User. OpenAI has a larger search and user share: GPTBot 69.7%, OAI-SearchBot 23.1%, ChatGPT-User 7.3%. Perplexity is almost all search crawling (PerplexityBot 92.3%). Google runs one mixed-use crawler for search and AI, which is why site owners cannot separate the two with a simple bot rule.

The bots you should know

Anthropic's help centre states that all three Claude bots honour robots.txt, support the non-standard Crawl-delay directive and do not bypass CAPTCHAs, and that blocking ClaudeBot excludes a site's future content from training. OpenAI publishes IP ranges for each bot (gptbot.json, searchbot.json, chatgpt-user.json) and notes that robots.txt updates take about 24 hours. For Manus, an agent that browses the web on a user's behalf, we found no separate published crawler token. Cloudflare's transparency table lists OpenAI, Anthropic, Meta and Amazon as verified and using distinct bots per purpose; Perplexity and ByteDance are not verified.

What websites block in robots.txt

Of the top 10,000 domains, 4,360 robots.txt files name at least one AI user agent. GPTBot is the most frequently blocked (437 domains fully, 197 partially), followed by ClaudeBot (390 / 172), CCBot (415 / 124) and Google-Extended (341 / 167). Search and user agents are blocked much less: OAI-SearchBot is fully disallowed on 118 domains, Googlebot on only 24. On Cloudflare's whole network, fewer than 1% of sites block search crawlers, while 17% block training.

A robots.txt policy for company websites

For a B2B company that wants to be found and cited in ChatGPT, Claude and Perplexity answers (see our guide to generative engine optimisation), the search and user agents are the ones that matter. Training is a separate decision, often driven by legal or licensing concerns (see AI copyright and training data).

• Always allow search and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) if you want citations and visits from AI answers.
• Decide on training per content type. Marketing pages can usually stay open; paid reports, documentation behind a login or licensed content can disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot.
• Do not block Googlebot or Bingbot to stop AI features; that removes you from search. Use Google-Extended and the providers' own snippet controls instead.
• Measure before you decide. Check your logs or CDN dashboard for which AI bots visit, which pages they read and which AI platforms send referrals.
• Enforce at the edge for bots that ignore robots.txt or are not verified: a WAF rule or your CDN's bot controls, not just a text file.
• Make your pages easy to read for agents. Clean HTML, a sitemap and fast responses help. Newer standards are still rare: in Cloudflare's scan of the top 200,000 domains, content signals appear on 5.3% of sites, Markdown negotiation on 4% and WebMCP on 1.3%.

What this means for companies in Croatia and the DACH region

For most companies in Croatia, Austria, Germany and Switzerland, the risk is not being crawled too much but being invisible in AI answers. A Croatian or German service page that blocks every AI bot will not be cited when a buyer asks Claude or ChatGPT for a supplier. At the same time, publishers, software vendors and anyone with paid content should treat training access as an asset under the EU copyright opt-out (Article 4 of the DSM Directive) and the GPAI Code of Practice, which commits signatories to honour robots.txt. The practical answer is the same for both: separate search from training, measure what the bots actually do and review the policy every quarter, because the numbers move fast.

Frequently asked questions

What is the crawl-to-refer ratio?

Cloudflare Radar compares how many HTML pages a platform's bots request with how many HTML page visits that platform refers to websites. A ratio of 487:1 means the platform's crawlers requested 487 pages for every visitor it sent back. In the four weeks to 27 October 2026 the ratio was about 4,000:1 for Perplexity, 881.5:1 for OpenAI, 486.8:1 for Anthropic and 4.7:1 for Google.

Should I block GPTBot and ClaudeBot?

It depends on whether you want your content used for model training. GPTBot and ClaudeBot are training crawlers; blocking them does not remove you from ChatGPT or Claude search, which use OAI-SearchBot and Claude-SearchBot. Many companies that sell services block or allow training by choice but always allow the search and user agents, because those bring citations and visitors.

Does blocking AI crawlers hurt my Google ranking?

Blocking Google-Extended stops use of your content for Gemini training and does not affect Google Search ranking, according to Google. Blocking Googlebot itself removes you from Google Search. Googlebot is a mixed-use crawler, so you cannot block AI Overviews and keep search with one rule; Google offers separate controls for that.

Do AI user agents like ChatGPT-User respect robots.txt?

OpenAI states that robots.txt rules may not apply to ChatGPT-User because its visits are initiated by a user. Anthropic says all three of its bots, including Claude-User, honour robots.txt. Training and search crawlers of the major providers follow robots.txt; for unverified bots only network-level blocking is reliable.