# AI crawlers and robots.txt: what online shops should allow

> The crawlers of OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Meta and Common Crawl, what each one does, and robots.txt rules for shops.

Source: https://central.to/guides/ai-crawlers-for-online-shops · Updated 2026-11-19 · Published by Central (https://central.to)

## In short

- AI companies run three kinds of bots: search crawlers that put pages into answers, training crawlers, and fetchers that visit a page when a user asks.
- Blocking GPTBot or ClaudeBot does not by itself remove a shop from ChatGPT or Claude search answers; those use OAI-SearchBot and Claude-SearchBot.
- Google-Extended and Applebot-Extended are control tokens, not crawlers; blocking them does not affect Google Search or Apple's search results.
- User-triggered fetchers such as ChatGPT-User, Perplexity-User and meta-externalfetcher may not follow robots.txt.
- A shop that wants to be found should allow the search and shopping crawlers and decide on training separately.

Free tool: [AI crawler access checker](https://central.to/tools/ai-crawler-checker). Check whether ChatGPT, Perplexity, Google's AI, Claude and other AI crawlers may read your shop.

## Three kinds of AI bots

AI companies that read the web use some mix of three kinds of visitors, and each one deserves its own robots.txt decision.

| Kind | What it does | Examples |
| --- | --- | --- |
| Search crawler | Indexes pages so an assistant can show and link them in answers | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `meta-webindexer` |
| Training crawler or token | Collects content, or controls whether crawled content may be used, to train models | `GPTBot`, `ClaudeBot`, `CCBot`, `meta-externalagent`, `Google-Extended`, `Applebot-Extended` |
| User-triggered fetcher | Visits one page because a person asked the assistant to | `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `meta-externalfetcher`, `Google-Agent` |

For a shop, search crawlers decide whether products can show up in assistant answers. Training crawlers and tokens decide whether your pages may help train future models. Fetchers act for a person who asked about your page.

## The crawlers, company by company

| Company | Token | What it does | robots.txt |
| --- | --- | --- | --- |
| OpenAI | `OAI-SearchBot` | Shows sites in ChatGPT's search features | Opted-out sites are left out of search answers but can still appear as navigational links |
| OpenAI | `GPTBot` | Crawls content that may train OpenAI's foundation models | Disallow it to keep content out of training |
| OpenAI | `ChatGPT-User` | Visits pages when a user asks ChatGPT or a custom GPT | Rules may not apply |
| Google | `Googlebot` | Crawls for Google Search | Followed |
| Google | `Storebot-Google` | Its rules apply to all Google Shopping surfaces | Followed |
| Google | `Google-Extended` | Token only, no own crawler; controls use for Gemini training and grounding | No effect on Search inclusion or ranking |
| Google | `Google-Agent` | Lets agents on Google infrastructure browse and act on a user's request | User-triggered, generally ignores robots.txt |
| Microsoft | `bingbot` | Bing's standard crawler | Followed |
| Perplexity | `PerplexityBot` | Surfaces and links sites in Perplexity results; not used to train foundation models | Perplexity asks you to allow it |
| Perplexity | `Perplexity-User` | Visits pages to answer a user's question | Generally ignored |
| Anthropic | `Claude-SearchBot` | Indexes pages to improve search results | Followed |
| Anthropic | `ClaudeBot` | Collects content that may be used for training | Followed; blocking signals that future content should stay out of training |
| Anthropic | `Claude-User` | Fetches pages when users ask Claude | Followed; blocking may reduce visibility in user searches |
| Apple | `Applebot` | Search in Spotlight, Siri and Safari; data may also train Apple's models | Followed; without Applebot rules it follows your Googlebot rules |
| Apple | `Applebot-Extended` | Token only, does not crawl; limits use for training Apple's models | Content can still appear in search |
| Meta | `meta-webindexer` | Improves Meta AI search results | Meta says allowing it helps cite and link you |
| Meta | `meta-externalagent` | Training AI models or indexing content for products | Block it with a Disallow rule |
| Meta | `meta-externalfetcher` | Fetches links at a user's request | May bypass robots.txt |
| Common Crawl | `CCBot` | Builds an open crawl dataset anyone can analyze | Disallow it to stop crawling |

Google's common crawlers always obey robots.txt when crawling automatically, while its [user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), such as `Google-Agent`, generally ignore it because a user requested the fetch.

Microsoft's crawler list has no separate AI training token. The [list](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) names `bingbot`, `AdIdxBot` for ads, and preview crawlers. Copilot use is controlled with meta tags instead: Bing's [Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a) say `NOARCHIVE` keeps content out of Copilot answers and grounding, and `NOCACHE` limits Copilot to the URL, title and snippet. Meta's crawlers are described on its [web crawlers page](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers); Common Crawl documents `CCBot` on its [CCBot page](https://commoncrawl.org/ccbot).

## A robots.txt for an online shop

This example lets search and shopping crawlers read the catalog, keeps all bots out of cart and account pages, and blocks model training. Northpeak, an outdoor gear shop, would put it at `https://example.com/robots.txt`. The first group applies to every bot not named elsewhere, including `Googlebot`, `Storebot-Google`, `bingbot`, `OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, `Applebot` and `meta-webindexer`. The second group blocks the training crawlers and tokens.

```
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
```

Three rules make this work:

1. **A crawler follows only one group.** Google's [robots.txt documentation](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) says a crawler uses the group with the most specific user agent that matches it and ignores the others. A bot named in its own group does not also read the `*` rules, so repeat any private paths there.
2. **Several user agents can share one group.** Repeating `User-agent:` lines before the rules applies them to each bot listed.
3. **Allowing training is a separate choice.** To let your pages train models, delete the second group. OpenAI, Google, Anthropic and Apple document their search crawlers separately from their training crawlers and tokens, and Perplexity says PerplexityBot is not used for training.

If you would rather name the search crawlers explicitly, give each one its own group with the same `Disallow` lines for private paths.

## Check before you block

- **Merchant Center fetches.** If Google fetches your product file on a schedule, Google [asks you](https://support.google.com/merchants/answer/1219255) not to block `Googlebot` and `AdsBot-Google` from the directory that holds it.
- **Subdomains count separately.** Anthropic reminds site owners to add rules for every subdomain they want to opt out.
- **Changes take time.** OpenAI says search changes take about 24 hours; Meta allows up to 24 hours.
- **User agents can be faked.** OpenAI publishes IP ranges for its bots, and Bing offers a tool to verify Bingbot. Check before you allow or block by IP.
- **Firewalls can block shoppers.** A bot rule on your CDN can stop ChatGPT's cloud browser from finishing a purchase for a customer. OpenAI says [Akamai, Cloudflare, HUMAN and Vercel](https://developers.openai.com/commerce/guides/enable-browser-access) verify its signed browser requests automatically.

A shared signal against AI training is arriving. Since 30 October 2026, Anthropic recognizes a machine-readable signal that lets site owners say "no AI training"; it follows a draft of the IETF aipref working group, and rules that name `ClaudeBot` take precedence. Google says it intends to support the shared AIPREF signal in early 2027. Until then, `Google-Extended` is the way to opt out of Gemini training.

## Where Central fits

Central publishes to the channels you choose. Every value is backed by sources you can check; what the sources do not support is held back.

## Questions

### If I block GPTBot, will my products disappear from ChatGPT?

Not because of GPTBot alone. OpenAI treats its bots independently: GPTBot covers model training, while OAI-SearchBot decides whether your site appears in ChatGPT's search answers.

### Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini training and grounding.

### Can I stop every AI visit with robots.txt?

No. Fetchers that act on a user's request, such as ChatGPT-User, Perplexity-User and meta-externalfetcher, say robots.txt may not apply to them. Anthropic says all of its bots honor robots.txt.

## Sources

1. [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots), checked 2026-11-19
2. [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), checked 2026-11-19
3. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked 2026-11-19
4. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots), checked 2026-11-19
5. [About Applebot](https://support.apple.com/en-us/119829), checked 2026-11-19
