GPTBot, ClaudeBot, PerplexityBot: The AI Crawler Decision Every Site Owner Faces

Every AI company runs crawlers, and your robots.txt is the policy layer that decides which ones may read you. For publishers who sell content, blocking is a defensible business decision. For brands that want to be recommended, blocking the wrong bot is self-inflicted invisibility. The trick is that the bots serve different purposes, and the right policy treats them differently.

The two kinds of AI crawler

Training crawlers collect content for model training. GPTBot (OpenAI) and ClaudeBot (Anthropic) are the prominent examples. Blocking them keeps your content out of future training corpora, which mainly affects what the next generation of models remembers about you.

Retrieval crawlers and fetchers feed live answers. OAI-SearchBot supports ChatGPT's search features, PerplexityBot builds Perplexity's index, and various user-triggered fetchers pull a page when an engine cites it in real time. Google's situation is the sharpest: its AI features draw on the same crawl as search, and Google-Extended controls only a slice of AI usage, so cutting Google off entirely is rarely on the table for anyone who wants organic traffic.

The brand calculus

For a brand, the question is not "do we want AI companies reading our site" but "when a buyer asks an engine about our category, do we want to be part of the answer." Framed that way:

• Allow retrieval bots, always. OAI-SearchBot, PerplexityBot, and their peers are the pipes through which citations flow. Blocking them removes you from live answers, the most valuable surface in AI search.
• Allow training bots for your marketing site. Training memory is where category associations live. You want the next model generation to remember your brand, your positioning, and your category, in your words.
• Block selectively where content is the product. Proprietary research, paid content, and user data deserve protection. Block at the directory level rather than site-wide, so your public marketing surface stays legible.

Verification beats policy

Robots.txt is a request, not a lock, and misconfigurations are rampant. Three checks worth an hour:

• Read your actual robots.txt today. CMS platforms, CDNs, and security vendors ship AI-bot blocklists that get enabled without anyone deciding.
• Grep your server logs for the major bot user agents. Confirm the bots you allow are actually fetching, and at sane rates.
• Check your CDN's bot management settings separately. Plenty of sites allow a bot in robots.txt and then serve it a 403 from the edge, which is worse than blocking cleanly.

The quiet failure mode we find most often in audits is exactly that: a brand investing in AEO content while its own infrastructure turns the readers away at the door. Crawl access is the least glamorous item in the program and the first one to verify, alongside whether your pages render for bots at all.