GPTBot vs OAI-SearchBot: what to block

Updated 18 October 2026 · AI Fix Shop

Check your site free: Run the free AI readiness checker (no sign-up, about 20 seconds).

"Should I block AI bots?" is really three questions: do you want your content used for training, do you want to appear in AI search answers, and do you want AI assistants to open your pages when a user asks? Each has its own crawler, so you can say yes to one and no to another.

The main AI crawlers and what they do

A few notes. Google-Extended isn't a separate crawler; it's a robots.txt token that tells Google whether content fetched by Googlebot may be used for its Gemini models. Blocking it does not affect normal Google Search. Applebot-Extended works the same way for Apple. User-triggered fetchers act on behalf of a person, and OpenAI and Perplexity say robots.txt rules may not apply to those requests the way they do to automated crawling. Bot names change over time, so check each company's crawler documentation before you rely on a list, including this one.

The common mistake

Many sites block GPTBot and assume they've "left ChatGPT", or block it by accident and assume they're invisible. Neither is true. GPTBot is about training. If OAI-SearchBot is allowed and can reach your pages, ChatGPT search can still find and cite you. The reverse also catches people: a blanket User-agent: * / Disallow: / blocks the search bots too.

Copy-paste robots.txt for three common choices

A. Visible in AI search, opted out of training (most small businesses)

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

B. Fully open

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

C. Out of AI entirely

Disallow the search and training bots above. Expect to lose AI search citations and referral traffic. This can make sense for paywalled or licensed content, but it's rarely right for a business that wants customers.

How to decide

• You sell services or products: you almost always want the search bots in. AI answers increasingly send people to the businesses they cite.
• Your content is the product (courses, research, paid articles): blocking training bots is reasonable; think carefully before blocking search bots, since they can still send readers.
• You're unsure: start with option A. It's easy to change later.

robots.txt is not the whole story

robots.txt only asks. Your CDN, firewall or security plugin can block a bot that robots.txt allows, and many hosts now have separate switches for search, training and agent traffic. After editing robots.txt, check those settings and look in your logs for the bot names. Rules can take time to be picked up after a crawler next fetches the file.

Want to see how your current setup is read? The free checker lists which AI search bots your robots.txt allows and flags training-only blocks separately. For the bigger picture, read how to check if ChatGPT can see your website.

FAQ

If I block GPTBot, will ChatGPT stop recommending me?

Not because of that block. GPTBot is used for training. ChatGPT search relies on OAI-SearchBot, which you can allow separately.

Does blocking Google-Extended hurt my Google rankings?

Google says no. Google-Extended only controls whether content can be used for its Gemini models; it is not used for Search ranking.

Is robots.txt enough to stop AI crawlers?

It's a public request that reputable crawlers follow. To enforce a block, use firewall or CDN rules as well, and verify bots by their published IP ranges.

Check your site free: Run the free AI readiness checker (no sign-up, about 20 seconds).