Question 1

Which AI crawlers should I allow to get cited by ChatGPT and Perplexity?

Accepted Answer

Allow OAI-SearchBot (ChatGPT Search), Claude-SearchBot (Claude), and PerplexityBot (Perplexity), plus Googlebot and Bingbot — Bing’s index feeds Microsoft Copilot and is a major source for ChatGPT. These answer-engine and search bots are separate from model-training crawlers, so you can be citable without opening your content to training.

Question 2

What is the difference between GPTBot and OAI-SearchBot?

Accepted Answer

GPTBot collects data to train OpenAI models. OAI-SearchBot surfaces and cites pages inside ChatGPT Search. Blocking GPTBot does not remove you from ChatGPT Search — that is controlled by OAI-SearchBot. Treat training and answer-engine bots as separate decisions.

Question 3

Do AI crawlers execute JavaScript?

Accepted Answer

Mostly no. Independent measurement of hundreds of millions of fetches found that GPTBot, ClaudeBot, and PerplexityBot do not render JavaScript. Content that only appears after client-side rendering is effectively invisible to them, so server-render or pre-render anything you want read and cited.

Question 4

How do I allow AI search but block AI model training?

Accepted Answer

Disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, CCBot) while allowing the answer-engine tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Use the "allow search, block training" snippet above. Note that some crawlers, such as Bytespider, are reported to ignore robots.txt and may need blocking at the network edge.