KhaqanWhat is robots.txt for AI crawlers?
robots.txt for AI crawlers means adding User-agent rules to the standard robots.txt file at a site's root that allow or disallow named AI bots individually, so a site can, for example, refuse model training while still allowing AI search engines to index it.
robots.txt is a plain text file at /robots.txt that well-behaved crawlers read before fetching anything else. It lists groups of rules, each starting with a User-agent line that names a bot (or an asterisk for all), followed by Disallow and Allow lines with path prefixes. The AI companies publish the user agent tokens their crawlers use, so a site can address each one by name.
It matters because the defaults are all-or-nothing. With no rules, every crawler is welcome, including training bots. With a single Disallow of the root under the wildcard user agent, everything is refused, including Googlebot and the AI search bots that would have cited you. The useful position for most businesses sits between those: allow the search and user-fetch agents, decide deliberately about the training agents, and never block Googlebot or Bingbot by accident.
A worked example for a company that wants AI search visibility but not training use: one group with User-agent: GPTBot and Disallow: /, another with User-agent: ClaudeBot and Disallow: /, and separate groups for OAI-SearchBot, Claude-SearchBot and PerplexityBot with Allow: /. Leave Google-Extended out unless you want to opt out of Gemini training, and remember that it has no effect on Google Search either way.
Two cautions. First, robots.txt is a request, not a lock; a crawler that ignores it can still fetch the page, and enforcement needs the firewall. Second, a firewall or CDN rule that blocks AI bots as a category usually blocks the search agents too, and the site then vanishes from answers while its robots.txt looks fine. That is why ai-visibility-audit refetches the homepage as GPTBot, PerplexityBot and ClaudeBot. The common mistake is copying a block list written for news publishers and applying it to a business that depends on being found.
Related terms
- AI crawler (GPTBot, ClaudeBot, PerplexityBot and others): An AI crawler is a bot that fetches web pages for an AI vendor to train models, to build a search index for an assistant, or to fetch a page live when a user asks; most vendors use a separate user agent for each purpose.
- GEO (generative engine optimisation): Generative engine optimisation (GEO) is the practice of making a website easy for generative AI systems such as ChatGPT, Perplexity, Claude and Gemini to crawl, understand and cite, so that the site appears as a source in AI-generated answers.
- llms.txt: llms.txt is a plain Markdown file placed at a website's root (/llms.txt) that gives AI assistants a short summary of the site and a curated list of its most useful pages, so a language model can find the right content quickly.