An AI crawler is an HTTP user agent operated by an AI company to fetch web pages for either model training or serving fresh grounded answers.
An AI crawler is an HTTP user agent operated by an AI company to fetch web pages for either model training or serving fresh grounded answers.
Named examples include OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; Anthropic's ClaudeBot; Perplexity's PerplexityBot and Perplexity-User; and Google's Google-Extended.
See: What is an AI crawler? and How should robots.txt handle AI crawlers?
An AI crawler is an HTTP user agent operated by an AI company to fetch web pages for either training a model or serving fresh grounded answers to end users — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended are the primary named examples in 2026.
Handle AI crawlers in robots.txt by explicitly listing each user agent — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended — and setting Allow or Disallow per bot to match your policy on training, search citation, and live-fetch use.
Block GPTBot only if you specifically want to opt out of OpenAI training use — blocking it does not affect ChatGPT Search citation eligibility, which is controlled by the separate OAI-SearchBot and ChatGPT-User user agents.
ChatGPT's crawl frequency depends on which of OpenAI's three user agents is fetching — GPTBot crawls on OpenAI's own schedule for training, OAI-SearchBot crawls to keep the ChatGPT Search index fresh (accelerated by IndexNow via Bing), and ChatGPT-User only fetches when a specific user prompt triggers a live retrieval.