← HomeHow to Get Cited in AI
Answer

What is an AI crawler?

By Jason Burns, Founder of HurcuLeads · Stuff Doer at Adolicious · Updated

An AI crawler is an HTTP user agent operated by an AI company to fetch web pages for either training a model or serving fresh grounded answers to end users — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended are the primary named examples in 2026.

Each has a distinct purpose. Training crawlers (GPTBot, ClaudeBot, Google-Extended) build the corpora used for base-model training. Search/citation crawlers (OAI-SearchBot, PerplexityBot) fetch pages for the retrieval indexes that power grounded, cited answers. Live-fetch agents (ChatGPT-User, Perplexity-User) only fire when a user prompt triggers a specific URL fetch.

All identify themselves via a standard User-Agent header and honor robots.txt — the RFC 9309 Robots Exclusion Protocol applies. Server operators can see them in access logs and permit or block them per agent.

Two operational facts. First, most AI crawlers do not execute JavaScript — a client-only page returns an empty shell to them. Second, the AI companies typically publish IP ranges and reverse-DNS patterns so publishers can verify that a claimed user agent is really the stated crawler and not a masquerading scraper.

Treat AI crawlers as first-class citizens in your crawl policy — allow or deny deliberately, don't ignore them.

Related terms

Sources

Related questions people ask next

  • How should robots.txt handle AI crawlers?

    Handle AI crawlers in robots.txt by explicitly listing each user agent — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended — and setting Allow or Disallow per bot to match your policy on training, search citation, and live-fetch use.

  • Does server-side rendering matter for AI crawlers?

    Server-side rendering matters for AI crawlers because most of them fetch a page's raw HTML once and do not execute JavaScript — a client-rendered page returns an empty shell on that fetch, so the answer text is invisible to the crawler even when the site works perfectly for a browser.

  • Should I block GPTBot?

    Block GPTBot only if you specifically want to opt out of OpenAI training use — blocking it does not affect ChatGPT Search citation eligibility, which is controlled by the separate OAI-SearchBot and ChatGPT-User user agents.

  • How often does ChatGPT re-crawl my site?

    ChatGPT's crawl frequency depends on which of OpenAI's three user agents is fetching — GPTBot crawls on OpenAI's own schedule for training, OAI-SearchBot crawls to keep the ChatGPT Search index fresh (accelerated by IndexNow via Bing), and ChatGPT-User only fetches when a specific user prompt triggers a live retrieval.

← All answersGlossaryReceipts