Skip to content

Guides AI crawlers and robots.txt: what online shops should allow

AI crawlers and robots.txt: what online shops should allow

The crawlers of OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Meta and Common Crawl, what each one does, and robots.txt rules for shops.

Updated 19 November 20265 min readDeutsch
In short
  • AI companies run three kinds of bots: search crawlers that put pages into answers, training crawlers, and fetchers that visit a page when a user asks.
  • Blocking GPTBot or ClaudeBot does not by itself remove a shop from ChatGPT or Claude search answers; those use OAI-SearchBot and Claude-SearchBot.
  • Google-Extended and Applebot-Extended are control tokens, not crawlers; blocking them does not affect Google Search or Apple's search results.
  • User-triggered fetchers such as ChatGPT-User, Perplexity-User and meta-externalfetcher may not follow robots.txt.
  • A shop that wants to be found should allow the search and shopping crawlers and decide on training separately.

Three kinds of AI bots#

AI companies that read the web use some mix of three kinds of visitors, and each one deserves its own robots.txt decision.

KindWhat it doesExamples
Search crawlerIndexes pages so an assistant can show and link them in answersOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer
Training crawler or tokenCollects content, or controls whether crawled content may be used, to train modelsGPTBot, ClaudeBot, CCBot, meta-externalagent, Google-Extended, Applebot-Extended
User-triggered fetcherVisits one page because a person asked the assistant toChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Google-Agent

For a shop, search crawlers decide whether products can show up in assistant answers. Training crawlers and tokens decide whether your pages may help train future models. Fetchers act for a person who asked about your page.

The crawlers, company by company#

CompanyTokenWhat it doesrobots.txt
OpenAIOAI-SearchBotShows sites in ChatGPT's search featuresOpted-out sites are left out of search answers but can still appear as navigational links
OpenAIGPTBotCrawls content that may train OpenAI's foundation modelsDisallow it to keep content out of training
OpenAIChatGPT-UserVisits pages when a user asks ChatGPT or a custom GPTRules may not apply
GoogleGooglebotCrawls for Google SearchFollowed
GoogleStorebot-GoogleIts rules apply to all Google Shopping surfacesFollowed
GoogleGoogle-ExtendedToken only, no own crawler; controls use for Gemini training and groundingNo effect on Search inclusion or ranking
GoogleGoogle-AgentLets agents on Google infrastructure browse and act on a user's requestUser-triggered, generally ignores robots.txt
MicrosoftbingbotBing's standard crawlerFollowed
PerplexityPerplexityBotSurfaces and links sites in Perplexity results; not used to train foundation modelsPerplexity asks you to allow it
PerplexityPerplexity-UserVisits pages to answer a user's questionGenerally ignored
AnthropicClaude-SearchBotIndexes pages to improve search resultsFollowed
AnthropicClaudeBotCollects content that may be used for trainingFollowed; blocking signals that future content should stay out of training
AnthropicClaude-UserFetches pages when users ask ClaudeFollowed; blocking may reduce visibility in user searches
AppleApplebotSearch in Spotlight, Siri and Safari; data may also train Apple's modelsFollowed; without Applebot rules it follows your Googlebot rules
AppleApplebot-ExtendedToken only, does not crawl; limits use for training Apple's modelsContent can still appear in search
Metameta-webindexerImproves Meta AI search resultsMeta says allowing it helps cite and link you
Metameta-externalagentTraining AI models or indexing content for productsBlock it with a Disallow rule
Metameta-externalfetcherFetches links at a user's requestMay bypass robots.txt
Common CrawlCCBotBuilds an open crawl dataset anyone can analyzeDisallow it to stop crawling

Google's common crawlers always obey robots.txt when crawling automatically, while its user-triggered fetchers, such as Google-Agent, generally ignore it because a user requested the fetch.

Microsoft's crawler list has no separate AI training token. The list names bingbot, AdIdxBot for ads, and preview crawlers. Copilot use is controlled with meta tags instead: Bing's Webmaster Guidelines say NOARCHIVE keeps content out of Copilot answers and grounding, and NOCACHE limits Copilot to the URL, title and snippet. Meta's crawlers are described on its web crawlers page; Common Crawl documents CCBot on its CCBot page.

A robots.txt for an online shop#

This example lets search and shopping crawlers read the catalog, keeps all bots out of cart and account pages, and blocks model training. Northpeak, an outdoor gear shop, would put it at https://example.com/robots.txt. The first group applies to every bot not named elsewhere, including Googlebot, Storebot-Google, bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot and meta-webindexer. The second group blocks the training crawlers and tokens.

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Three rules make this work:

  1. A crawler follows only one group. Google's robots.txt documentation says a crawler uses the group with the most specific user agent that matches it and ignores the others. A bot named in its own group does not also read the * rules, so repeat any private paths there.
  2. Several user agents can share one group. Repeating User-agent: lines before the rules applies them to each bot listed.
  3. Allowing training is a separate choice. To let your pages train models, delete the second group. OpenAI, Google, Anthropic and Apple document their search crawlers separately from their training crawlers and tokens, and Perplexity says PerplexityBot is not used for training.

If you would rather name the search crawlers explicitly, give each one its own group with the same Disallow lines for private paths.

Check before you block#

  • Merchant Center fetches. If Google fetches your product file on a schedule, Google asks you not to block Googlebot and AdsBot-Google from the directory that holds it.
  • Subdomains count separately. Anthropic reminds site owners to add rules for every subdomain they want to opt out.
  • Changes take time. OpenAI says search changes take about 24 hours; Meta allows up to 24 hours.
  • User agents can be faked. OpenAI publishes IP ranges for its bots, and Bing offers a tool to verify Bingbot. Check before you allow or block by IP.
  • Firewalls can block shoppers. A bot rule on your CDN can stop ChatGPT's cloud browser from finishing a purchase for a customer. OpenAI says Akamai, Cloudflare, HUMAN and Vercel verify its signed browser requests automatically.

A shared signal against AI training is arriving. Since 30 October 2026, Anthropic recognizes a machine-readable signal that lets site owners say "no AI training"; it follows a draft of the IETF aipref working group, and rules that name ClaudeBot take precedence. Google says it intends to support the shared AIPREF signal in early 2027. Until then, Google-Extended is the way to opt out of Gemini training.

Where Central fits#

Central publishes to the channels you choose. Every value is backed by sources you can check; what the sources do not support is held back.

Questions

If I block GPTBot, will my products disappear from ChatGPT?

Not because of GPTBot alone. OpenAI treats its bots independently: GPTBot covers model training, while OAI-SearchBot decides whether your site appears in ChatGPT's search answers.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini training and grounding.

Can I stop every AI visit with robots.txt?

No. Fetchers that act on a user's request, such as ChatGPT-User, Perplexity-User and meta-externalfetcher, say robots.txt may not apply to them. Anthropic says all of its bots honor robots.txt.

Sources

  1. Overview of OpenAI Crawlers · OpenAI developer docs checked 19 November 2026
  2. Google's common crawlers · Google for Developers checked 19 November 2026
  3. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic checked 19 November 2026
  4. Perplexity crawlers · Perplexity checked 19 November 2026
  5. About Applebot · Apple checked 19 November 2026

You choose what goes in and where it goes out.

Central AI does everything in between. Your first 3 products are free.

  1. Your inputYour shop, your files or your product pages.
  2. Central AIResearches, verifies and writes everything in between. Every value is backed by sources you can check; what the sources do not support is held back.
  3. Your channelsProduct data and listings for the channels you choose.