Live on Product Hunt Intraface launched today. Less steering, more vacations. See the launch
Start for free

What are AI crawlers?

AI crawlers are the automated fetchers that AI companies run against the web: GPTBot and ClaudeBot gather training data, OAI-SearchBot, Claude-SearchBot and PerplexityBot build search indexes, and ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks about it. Each announces itself in its user agent, and robots.txt rules are read per bot, though user-triggered fetchers may ignore them.

Part of Glossary: the terms behind MCP, coding agents and shared pages, defined plainly.

How it works

Three kinds of AI bot fetch your site for three reasons, and robots.txt lets you allow or block each one by name.training crawlerGPTBot, ClaudeBotsearch crawlerOAI-SearchBot and othersuser fetcherChatGPT-User, Claude-Userrobots.txt,one rule per botyour siteuser fetchers mayskip robots.txtfor training datafor a search indexone page, on requestallow or block
  1. A crawler requests a page with a user agent that names it, such as GPTBot/1.4 or ClaudeBot, and links to its documentation.

  2. Before crawling, it reads /robots.txt and obeys rules addressed to its name. Google-Extended is a robots.txt token only; no request carries it as a user agent.

  3. Training crawlers (GPTBot, ClaudeBot) feed model training. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build a search index.

  4. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page because someone asked about it. Two vendors say robots.txt may not apply to them.

  5. Each vendor publishes its crawler IP ranges as JSON (for example openai.com/gptbot.json and claude.com/crawling/bots.json) so you can check a bot is real.

Source: OpenAI: Overview of OpenAI crawlers

The three kinds of AI crawler, and why the names confuse people

Each AI company runs more than one bot, and the one people have heard of is usually not the one that decides whether they show up in answers. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for pages a user asks about during a chat. Anthropic documents ClaudeBot for model development, Claude-SearchBot for its search results, and Claude-User for user-requested fetches. Perplexity documents PerplexityBot for its search index and Perplexity-User for live fetches, and says plainly that neither is used to train foundation models.

The split matters because robots.txt rules are read per user agent. A line that disallows GPTBot says "don't train on this" and nothing about search. Google's equivalent, Google-Extended, is odder still: it is a robots.txt token with no user agent of its own, it controls whether crawled content is used for Gemini training and grounding, and Google says it has no effect on inclusion or ranking in Search.

The user fetchers are the exception to the per-bot control. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person triggered the request, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says its bots honour robots.txt and lists Claude-User among the bots you can disable by name. If a page is public, assume a user can have an assistant read it.

Example: intraface.cc's robots.txt

Intraface's site takes the open position. Its robots.txt is User-agent: * followed by Allow: /, then the sitemap lines, so every bot in the lists above may fetch every public page. The reasoning is that the site exists to be read, and the answer-first pages and JSON-LD on it only pay off if the search crawlers can reach them. What should not be crawled is handled on the page instead: noindex on the 404 page and sign-in shims, and a separate noindex domain for the boards users publish. Allowing crawlers and controlling indexing are different levers, and the site uses the second rather than the first.

Why it matters

Whether an assistant can cite you is decided before any content question: did its search crawler get in. Many sites added a blanket block on everything with "GPT" or "AI" in the name in 2023 and have never revisited it. Those sites are absent from ChatGPT search and Perplexity today without knowing why. Conversely, a site that wants to keep its text out of training can do that precisely, by name, without giving up search visibility.

Common mistakes

  • Blocking GPTBot to stay out of ChatGPT. That stops training. OAI-SearchBot is what puts you in ChatGPT search answers.
  • Writing a Google-Extended user agent rule and expecting a bot to show up with it. Nothing does; it only works as a robots.txt token.
  • Blocking by IP first. Anthropic warns that IP blocks can stop the bot from reading robots.txt at all, so the opt-out never registers. Use robots.txt, then the IP lists for verification.
  • Trusting the user agent string. It is text anyone can send. Verify against the published IP JSON.
  • Expecting robots.txt to stop live fetches. Two vendors say their user-triggered fetchers may not honour it.

FAQ

Should I block AI crawlers in robots.txt?
Decide per bot, because they do different jobs. Disallowing GPTBot or ClaudeBot opts your future content out of training and nothing else. Disallowing OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from that vendor's search answers. Most sites that want AI visibility block the training bots, if any, and allow the search bots.
How do I verify an AI crawler is genuine?
Check the source IP against the vendor's published list: OpenAI at openai.com/gptbot.json, searchbot.json and chatgpt-user.json, Anthropic at claude.com/crawling/bots.json, Perplexity at perplexity.com/perplexitybot.json and perplexity-user.json. A user agent string alone proves nothing, since anyone can send one.
Do AI crawlers respect Crawl-delay?
Anthropic says it supports the Crawl-delay extension to robots.txt for ClaudeBot and applies it where appropriate. OpenAI's and Perplexity's crawler pages don't mention the directive. If a bot is hitting you too hard, the documented levers are robots.txt rules and, as a last resort, the published IP ranges.

Stop sending HTMLover Slack.

Create an account in a minute, connect your coding agent, and make your first board.