# AI crawler (bots that read sites for AI)

> Which AI crawlers visit online shops, what each one is used for, how to allow or block them in robots.txt, and why search and training crawlers differ.

Source: https://central.to/glossary/ai-crawler · Updated 2026-11-19 · Published by Central (https://central.to)

**Definition:** An AI crawler is an automated client that fetches web pages for an AI company, to train models, to show sites in AI search answers, or to open a page a user asked an assistant about.

Free tool: [AI crawler access checker](https://central.to/tools/ai-crawler-checker). Check whether ChatGPT, Perplexity, Google's AI, Claude and other AI crawlers may read your shop.

## Three jobs, different bots

AI companies fetch pages for different purposes, and each purpose has its own name in `robots.txt`.

| Name | Company | Purpose |
| --- | --- | --- |
| OAI-SearchBot | OpenAI | Shows websites in ChatGPT's search features |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT or a custom GPT; not automatic crawling |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ads in ChatGPT |
| Google-Extended | Google | Controls whether crawled content may be used for Gemini model training and grounding |

## Search is not training

OpenAI treats each setting independently: a site can allow OAI-SearchBot to appear in ChatGPT search and disallow GPTBot to keep its content out of training. A site that opts out of OAI-SearchBot is not shown in ChatGPT search answers, though it can still appear as a navigational link. OpenAI says robots.txt changes for search can take about 24 hours to apply.

Because ChatGPT-User acts on a user's request, OpenAI says robots.txt rules may not apply to it.

Google-Extended is a product token, not a separate crawler: Google crawls with its usual user agents and reads the `Google-Extended` group in robots.txt as a control. Google says it does not affect inclusion or ranking in Google Search.

## Example robots.txt

```
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

This allows OAI-SearchBot for ChatGPT search, signals that the pages should not be used to train OpenAI's models, and opts the content out of Gemini training and grounding through Google-Extended.

## Questions

### If I block GPTBot, do my products disappear from ChatGPT search?

No. OpenAI treats each setting separately. GPTBot is for model training; OAI-SearchBot decides whether a site appears in ChatGPT's search answers.

### Does blocking Google-Extended affect Google Search?

No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.

## Sources

1. [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots), checked 2026-11-19
2. [List of Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), checked 2026-11-19
