WEBSITE GUIDE

ChatGPT search and AI training use different crawlers

By IDAKIM Digital · Published November 6, 2026

OpenAI documents separate controls for search visibility and model training. Website owners can allow OAI-SearchBot while disallowing GPTBot.

OpenAI’s crawler documentation identifies OAI-SearchBot as its search crawler and GPTBot as the crawler for content that may be used to train foundation models. Their robots.txt settings are independent. ChatGPT-User serves user-initiated visits, for which robots.txt rules may not apply. OpenAI says search systems can take approximately 24 hours to reflect a robots.txt update.

Why it matters

Decide separately whether you want search discovery and training crawling. Back up existing rules, review the relevant user-agent groups, and check the live file after publishing. A robots.txt instruction is not authentication or a guarantee against collection by other services.

Source: OpenAI · crawler documentation

Product claims are attributed to the linked publisher. IDAKIM Digital has not independently tested these claims.