Guides AI crawlers and robots.txt: what online shops should allow
AI crawlers and robots.txt: what online shops should allow
The crawlers of OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Meta and Common Crawl, what each one does, and robots.txt rules for shops.
- AI companies run three kinds of bots: search crawlers that put pages into answers, training crawlers, and fetchers that visit a page when a user asks.
- Blocking GPTBot or ClaudeBot does not by itself remove a shop from ChatGPT or Claude search answers; those use OAI-SearchBot and Claude-SearchBot.
- Google-Extended and Applebot-Extended are control tokens, not crawlers; blocking them does not affect Google Search or Apple's search results.
- User-triggered fetchers such as ChatGPT-User, Perplexity-User and meta-externalfetcher may not follow robots.txt.
- A shop that wants to be found should allow the search and shopping crawlers and decide on training separately.
Three kinds of AI bots#
AI companies that read the web use some mix of three kinds of visitors, and each one deserves its own robots.txt decision.
| Kind | What it does | Examples |
|---|---|---|
| Search crawler | Indexes pages so an assistant can show and link them in answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer |
| Training crawler or token | Collects content, or controls whether crawled content may be used, to train models | GPTBot, ClaudeBot, CCBot, meta-externalagent, Google-Extended, Applebot-Extended |
| User-triggered fetcher | Visits one page because a person asked the assistant to | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Google-Agent |
For a shop, search crawlers decide whether products can show up in assistant answers. Training crawlers and tokens decide whether your pages may help train future models. Fetchers act for a person who asked about your page.
The crawlers, company by company#
| Company | Token | What it does | robots.txt |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Shows sites in ChatGPT's search features | Opted-out sites are left out of search answers but can still appear as navigational links |
| OpenAI | GPTBot | Crawls content that may train OpenAI's foundation models | Disallow it to keep content out of training |
| OpenAI | ChatGPT-User | Visits pages when a user asks ChatGPT or a custom GPT | Rules may not apply |
Googlebot | Crawls for Google Search | Followed | |
Storebot-Google | Its rules apply to all Google Shopping surfaces | Followed | |
Google-Extended | Token only, no own crawler; controls use for Gemini training and grounding | No effect on Search inclusion or ranking | |
Google-Agent | Lets agents on Google infrastructure browse and act on a user's request | User-triggered, generally ignores robots.txt | |
| Microsoft | bingbot | Bing's standard crawler | Followed |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results; not used to train foundation models | Perplexity asks you to allow it |
| Perplexity | Perplexity-User | Visits pages to answer a user's question | Generally ignored |
| Anthropic | Claude-SearchBot | Indexes pages to improve search results | Followed |
| Anthropic | ClaudeBot | Collects content that may be used for training | Followed; blocking signals that future content should stay out of training |
| Anthropic | Claude-User | Fetches pages when users ask Claude | Followed; blocking may reduce visibility in user searches |
| Apple | Applebot | Search in Spotlight, Siri and Safari; data may also train Apple's models | Followed; without Applebot rules it follows your Googlebot rules |
| Apple | Applebot-Extended | Token only, does not crawl; limits use for training Apple's models | Content can still appear in search |
| Meta | meta-webindexer | Improves Meta AI search results | Meta says allowing it helps cite and link you |
| Meta | meta-externalagent | Training AI models or indexing content for products | Block it with a Disallow rule |
| Meta | meta-externalfetcher | Fetches links at a user's request | May bypass robots.txt |
| Common Crawl | CCBot | Builds an open crawl dataset anyone can analyze | Disallow it to stop crawling |
Google's common crawlers always obey robots.txt when crawling automatically, while its user-triggered fetchers, such as Google-Agent, generally ignore it because a user requested the fetch.
Microsoft's crawler list has no separate AI training token. The list names bingbot, AdIdxBot for ads, and preview crawlers. Copilot use is controlled with meta tags instead: Bing's Webmaster Guidelines say NOARCHIVE keeps content out of Copilot answers and grounding, and NOCACHE limits Copilot to the URL, title and snippet. Meta's crawlers are described on its web crawlers page; Common Crawl documents CCBot on its CCBot page.
A robots.txt for an online shop#
This example lets search and shopping crawlers read the catalog, keeps all bots out of cart and account pages, and blocks model training. Northpeak, an outdoor gear shop, would put it at https://example.com/robots.txt. The first group applies to every bot not named elsewhere, including Googlebot, Storebot-Google, bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot and meta-webindexer. The second group blocks the training crawlers and tokens.
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Three rules make this work:
- A crawler follows only one group. Google's robots.txt documentation says a crawler uses the group with the most specific user agent that matches it and ignores the others. A bot named in its own group does not also read the
*rules, so repeat any private paths there. - Several user agents can share one group. Repeating
User-agent:lines before the rules applies them to each bot listed. - Allowing training is a separate choice. To let your pages train models, delete the second group. OpenAI, Google, Anthropic and Apple document their search crawlers separately from their training crawlers and tokens, and Perplexity says PerplexityBot is not used for training.
If you would rather name the search crawlers explicitly, give each one its own group with the same Disallow lines for private paths.
Check before you block#
- Merchant Center fetches. If Google fetches your product file on a schedule, Google asks you not to block
GooglebotandAdsBot-Googlefrom the directory that holds it. - Subdomains count separately. Anthropic reminds site owners to add rules for every subdomain they want to opt out.
- Changes take time. OpenAI says search changes take about 24 hours; Meta allows up to 24 hours.
- User agents can be faked. OpenAI publishes IP ranges for its bots, and Bing offers a tool to verify Bingbot. Check before you allow or block by IP.
- Firewalls can block shoppers. A bot rule on your CDN can stop ChatGPT's cloud browser from finishing a purchase for a customer. OpenAI says Akamai, Cloudflare, HUMAN and Vercel verify its signed browser requests automatically.
A shared signal against AI training is arriving. Since 30 October 2026, Anthropic recognizes a machine-readable signal that lets site owners say "no AI training"; it follows a draft of the IETF aipref working group, and rules that name ClaudeBot take precedence. Google says it intends to support the shared AIPREF signal in early 2027. Until then, Google-Extended is the way to opt out of Gemini training.
Where Central fits#
Central publishes to the channels you choose. Every value is backed by sources you can check; what the sources do not support is held back.
Questions
If I block GPTBot, will my products disappear from ChatGPT?
Not because of GPTBot alone. OpenAI treats its bots independently: GPTBot covers model training, while OAI-SearchBot decides whether your site appears in ChatGPT's search answers.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini training and grounding.
Can I stop every AI visit with robots.txt?
No. Fetchers that act on a user's request, such as ChatGPT-User, Perplexity-User and meta-externalfetcher, say robots.txt may not apply to them. Anthropic says all of its bots honor robots.txt.
Sources
- Overview of OpenAI Crawlers · OpenAI developer docs checked 19 November 2026
- Google's common crawlers · Google for Developers checked 19 November 2026
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic checked 19 November 2026
- Perplexity crawlers · Perplexity checked 19 November 2026
- About Applebot · Apple checked 19 November 2026