← All articles

3 min read

Should You Let AI Crawlers Like GPTBot Read Your Site?

If you want AI assistants to find and cite your business, allow their search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude) and PerplexityBot (Perplexity). Training crawlers like GPTBot and ClaudeBot are a separate choice. Blocking them signals your content shouldn't be used to train future models, and the companies' docs say search access is controlled separately. You can block training and still allow search.

Training bots vs search bots

AI companies run more than one crawler. The difference matters:

  • Training crawlers collect pages that may be used to train future AI models.
  • Search crawlers index pages so the assistant can find and cite them when answering live questions.
  • User fetchers visit a page when a person’s request needs it, like “summarize this link”.

Blocking one doesn’t block the others. OpenAI says its settings are independent, so you can allow search and block training.

What each crawler does

Company Crawler Job What blocking it does (per the company)
OpenAI OAI-SearchBot Search in ChatGPT Your site is not shown in ChatGPT search answers, though it can still appear as a navigational link
OpenAI GPTBot Training Signals your content shouldn’t be used to train OpenAI’s models
OpenAI ChatGPT-User User requests OpenAI says robots.txt rules may not apply, because a user started the request
Anthropic Claude-SearchBot Search in Claude May reduce your visibility and accuracy in Claude’s search results
Anthropic ClaudeBot Training Signals future content should be excluded from training
Anthropic Claude-User User requests Stops Claude fetching your pages for users, which may reduce visibility
Perplexity PerplexityBot Search in Perplexity Perplexity recommends allowing it if you want to appear in results. It says this bot isn’t used for training
Perplexity Perplexity-User User requests Perplexity says it generally ignores robots.txt, since a user asked for the page

Sources: OpenAI, Anthropic, Perplexity.

What about Google?

Google works differently. Its AI Overviews and AI Mode are part of Google Search, so Googlebot’s robots.txt rules are the control. Block Googlebot and you leave Search entirely.

Google has two narrower options:

  • Google-Extended limits AI training and grounding in some of Google’s other systems. Google lists it separately from Search controls.
  • A Search Console setting lets you opt out of Google’s generative AI Search features. Google says sites that opt out get no traffic or impressions from those features, and the setting isn’t used as a ranking signal for regular results. Google says it rolled this out to all sites worldwide by August 31, 2026.

If you want to appear in AI Overviews and AI Mode, check that this setting is on.

A robots.txt example: allow search, block training

This is one possible setup. It’s a choice, not a requirement.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

If you’re happy for your content to be used in training too, you can allow everything. A local service business with public prices and service pages may not lose much by allowing training. A publisher or course creator selling content may see it differently.

Check for hidden blocks

Your robots.txt might be fine while something else blocks the bots:

  • Firewalls and CDNs. If your host or CDN has bot protection turned on, it may block these crawlers. OpenAI and Perplexity both recommend allowing their published IP ranges.
  • IP blocking. Anthropic says blocking its IP addresses may not work reliably as an opt out, because it stops the bot from reading your robots.txt. Use robots.txt instead.
  • Subdomains. Anthropic notes you need rules on every subdomain you want to cover.

How fast do changes apply?

OpenAI says robots.txt changes for search can take about 24 hours. Perplexity says up to 24 hours. That’s access only. Being named in answers depends on much more than access, so allowing a crawler doesn’t mean you’ll be cited the next day.

Where Meristoll fits

Crawler access is one of the first things our free report looks at, along with your website, listings, reviews and mentions. It’s also one of the faster fixes: letting AI crawlers read your site can work within days. Real movement in answers usually takes longer. You’ll get a 0 to 100 score for six assistants and a prioritized fix list.

Questions people ask

Will blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search, according to OpenAI. GPTBot is the training crawler, and search inclusion is controlled by OAI-SearchBot. The two settings are independent.

What is OAI-SearchBot?

It's the crawler OpenAI uses to surface websites in ChatGPT's search features. OpenAI says sites that block it won't be shown in ChatGPT search answers, though they can still appear as navigational links.

Does blocking AI crawlers protect my content?

Partly. Blocking training crawlers signals your content shouldn't be used for training, and the companies say they honor robots.txt. User fetchers like ChatGPT-User and Perplexity-User may still visit a page when a user asks, because the request comes from a person.

Does blocking Google-Extended remove me from AI Overviews?

No. Google lists Googlebot as the control for Search, including AI features. Google-Extended covers training and grounding in some of Google's other systems. To opt out of AI Overviews and AI Mode, Google points to the Search Console setting.

How do I check what my robots.txt says?

Open yourdomain.com/robots.txt in a browser. Look for the bot names in the table above and any "Disallow: /" lines under them, or under "User-agent: *", which applies to every bot without its own rule.

Sources

  1. OpenAI: overview of OpenAI crawlers
  2. Anthropic: how Anthropic crawls the web and how site owners can block it
  3. Perplexity: crawler documentation
  4. Google Search Central: AI features and your website
  5. Google blog: new controls for website owners

Does AI recommend your business?

Get a free report: what ChatGPT, Gemini and four other AI assistants tell your customers, and what to fix first.

Check if AI recommends you