Handle AI crawlers in robots.txt by explicitly listing each user agent — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended — and setting Allow or Disallow per bot to match your policy on training, search citation, and live-fetch use.
Each AI provider publishes the exact user-agent strings and their purposes. OpenAI's bots documentation distinguishes GPTBot (training), OAI-SearchBot (indexing for ChatGPT Search), and ChatGPT-User (live user-triggered fetch). Anthropic's crawler article covers ClaudeBot and related agents. Perplexity's bots guide covers PerplexityBot and Perplexity-User. Google's publisher-controls announcement introduces Google-Extended, which controls training use without affecting Googlebot indexing.
The syntax is standard RFC 9309. A publisher wanting to allow citation-oriented crawlers but block training might write per-agent User-agent: blocks with a mix of Allow: / and Disallow: / lines.
Two nuances. First, robots.txt is advisory — well-behaved crawlers respect it, malicious scrapers do not. Second, allowing the search-oriented crawler while disallowing the training crawler is legitimate under each provider's terms and gives publishers a clean way to be citable without contributing to model training.
Also list your Sitemap: directive — the same file is read by many AI crawlers for discovery.