Two crawlers doing two different jobs
OpenAI publishes a page describing its crawlers and the split is explicit. GPTBot is used to crawl content that may be used in training its generative AI foundation models. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
A third agent, ChatGPT-User, visits a page when somebody asks ChatGPT a question that needs it. OpenAI notes that ChatGPT-User is not used for crawling the web in an automatic fashion. So a brand that wants no part in training while staying visible in answers has a clean position available to it. Disallow GPTBot. Allow OAI-SearchBot. Those are two separate lines in one file.
Why so many brands blocked the wrong one
A great many sites blocked everything with an AI sounding name in 2023 and 2024, usually on legal advice about training data. The intention was sound. The effect on some of those sites was removal from ChatGPT’s search answers, which nobody in that meeting had asked for. Reading your own robots.txt file takes two minutes. It is the cheapest check in this entire subject and it is free.
Every other platform draws the same line. Anthropic documents ClaudeBot as the crawler that collects web content for its models, with Claude-SearchBot navigating the web to improve search result quality. Anthropic says that disabling Claude-SearchBot prevents its system from indexing your content, which may reduce your site’s visibility and accuracy in user search results. Perplexity documents PerplexityBot as designed to surface and link websites in search results on Perplexity while stating that it is not used to crawl content for AI foundation models. Perplexity recommends allowing it in robots.txt.
How to check your own setup in ten minutes
Open your own domain followed by /robots.txt and read it. Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Bingbot and Google-Extended. Google-Extended is the one people misread in both directions. It controls whether content is used to train Gemini models plus ground answers in Gemini apps. Google states that it does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.
Then check the layer almost nobody checks. Google’s AI features guidance tells site owners to allow crawling in robots.txt and by any CDN or host. Perplexity tells site owners to allow its published addresses through any firewall. OpenAI tells site owners to confirm that the website host or content delivery network allows traffic from its published searchbot addresses. A welcoming robots.txt file sitting behind a bot protection rule that blocks everything is a common invisible cause of a brand being absent from assistants. The fix is usually a rule a developer adds in minutes.