Glossary AI crawler
AI crawler (bots that read sites for AI)
Which AI crawlers visit online shops, what each one is used for, how to allow or block them in robots.txt, and why search and training crawlers differ.
An AI crawler is an automated client that fetches web pages for an AI company, to train models, to show sites in AI search answers, or to open a page a user asked an assistant about.
Three jobs, different bots#
AI companies fetch pages for different purposes, and each purpose has its own name in robots.txt.
| Name | Company | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Shows websites in ChatGPT's search features |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT or a custom GPT; not automatic crawling |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ads in ChatGPT |
| Google-Extended | Controls whether crawled content may be used for Gemini model training and grounding |
Search is not training#
OpenAI treats each setting independently: a site can allow OAI-SearchBot to appear in ChatGPT search and disallow GPTBot to keep its content out of training. A site that opts out of OAI-SearchBot is not shown in ChatGPT search answers, though it can still appear as a navigational link. OpenAI says robots.txt changes for search can take about 24 hours to apply.
Because ChatGPT-User acts on a user's request, OpenAI says robots.txt rules may not apply to it.
Google-Extended is a product token, not a separate crawler: Google crawls with its usual user agents and reads the Google-Extended group in robots.txt as a control. Google says it does not affect inclusion or ranking in Google Search.
Example robots.txt#
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This allows OAI-SearchBot for ChatGPT search, signals that the pages should not be used to train OpenAI's models, and opts the content out of Gemini training and grounding through Google-Extended.
Questions
If I block GPTBot, do my products disappear from ChatGPT search?
No. OpenAI treats each setting separately. GPTBot is for model training; OAI-SearchBot decides whether a site appears in ChatGPT's search answers.
Does blocking Google-Extended affect Google Search?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.
Sources
- Overview of OpenAI Crawlers · OpenAI checked 19 November 2026
- List of Google's common crawlers · Google for Developers checked 19 November 2026