KhaqanWhat is AI crawler (GPTBot, ClaudeBot, PerplexityBot and others)?
An AI crawler is a bot that fetches web pages for an AI vendor to train models, to build a search index for an assistant, or to fetch a page live when a user asks; most vendors use a separate user agent for each purpose.
The names matter because each one controls something different. For OpenAI, GPTBot gathers pages for training, OAI-SearchBot builds the index behind ChatGPT search, and ChatGPT-User fetches a page when a person asks the assistant to look at it. Anthropic mirrors this with ClaudeBot for training, Claude-SearchBot for its search index and Claude-User for user-initiated fetches. Perplexity uses PerplexityBot for its index and Perplexity-User for live fetches. Google's search products, including AI Overviews, use the ordinary Googlebot crawl; Google-Extended is a robots.txt token rather than a separate crawler, and it only governs whether crawled content may train or ground Gemini. Bingbot's index powers Copilot and is one of the sources behind ChatGPT search. CCBot belongs to Common Crawl, the non-profit archive whose data has trained many models. Apple, Meta, Amazon and ByteDance run crawlers of their own.
The distinction between training, indexing and user fetches is the whole point. A business may reasonably refuse to donate its content to model training while still wanting to be found and cited by AI search. That is only possible if the two are treated separately in robots.txt. User-initiated fetchers are a special case: because a person asked for the page, some vendors state that these agents do not follow robots.txt at all.
An example: a services company blocks GPTBot and ClaudeBot, allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, and leaves Google-Extended alone. It stays visible in every assistant's search results while opting out of training. The common mistake is the reverse: a blanket block on anything with bot in the name, usually via a CDN toggle, which removes the site from AI search in one click. The robots.txt table in ai-visibility-audit exists to make this visible.
Related terms
- robots.txt for AI crawlers: robots.txt for AI crawlers means adding User-agent rules to the standard robots.txt file at a site's root that allow or disallow named AI bots individually, so a site can, for example, refuse model training while still allowing AI search engines to index it.
- GEO (generative engine optimisation): Generative engine optimisation (GEO) is the practice of making a website easy for generative AI systems such as ChatGPT, Perplexity, Claude and Gemini to crawl, understand and cite, so that the site appears as a source in AI-generated answers.
- AI search visibility: AI search visibility is how often, and how accurately, a website is used and cited by AI-driven search products such as ChatGPT search, Perplexity, Google AI Overviews, Copilot and Claude when people ask questions the site could answer.