AI crawlers and robots.txt: which bots to allow

Why does robots.txt decide whether AI can cite my business?

Answer engines like ChatGPT, Perplexity and Claude send crawlers to read your pages, and those crawlers check robots.txt first — block them and your site can't be indexed, quoted or linked.

Robots.txt is a plain-text file at the root of your domain (yourcompany.com/robots.txt). It has been the web's "please don't crawl this" signal since Martijn Koster defined it in 1994, and it became a formal internet standard, RFC 9309, in September 2022. Every reputable crawler, including the AI ones, reads it before touching anything else.

That makes it the first gate in answer engine optimization. Great reviews, clean business facts and helpful service pages do nothing for AI visibility if the crawler that feeds an answer engine is turned away at the door.

The stakes are real. BrightLocal's 2026 survey of 1,002 US adults found 45% of consumers now use AI tools for local business recommendations, up from 6% the year before.

An analysis of 70,723 ChatGPT citations across 100 US and Canadian metros by Cheers found 48.8% pointed to business websites — the single biggest source type. Your own site is the thing most worth keeping open.

We see it in our own pipeline: many of the business owners who call Media Spearhead now say an AI assistant recommended us, and AI has become our biggest source of new clients. None of that happens if the crawlers can't read the site.

What is the difference between answer crawlers and training crawlers?

Answer crawlers fetch pages so an AI can search, cite and link them today; training crawlers collect content to build future models — blocking the first removes you from answers, blocking the second does not.

This is the idea most "block AI crawlers" advice gets wrong. The major AI companies now run separate bots for separate jobs, and each gets its own name in robots.txt. You can say yes to one and no to another.

There are three kinds worth knowing:

• Search and answer crawlers build the index an assistant searches when someone asks "best emergency plumber near me." Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot. Block these and you can't be cited.
• User-triggered fetchers open one page because a person asked the assistant to look at it or needed fresh information. Examples: ChatGPT-User, Perplexity-User, Claude-User. Several vendors say robots.txt may not apply to these, because a human initiated the request.
• Training crawlers and control tokens decide whether your content goes into the data used to train future models. Examples: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended. Blocking these does not remove you from live answers.

OpenAI states it plainly in its crawler documentation: GPTBot is used to make its foundation models "more useful and safe," while OAI-SearchBot is used "to surface websites in search results in ChatGPT's search features." A site can block GPTBot and still appear in ChatGPT search, as long as OAI-SearchBot is allowed.

Which AI crawlers should a local business allow or block?

Allow every search and answer crawler, always allow Googlebot and Bingbot, and treat training-only crawlers as an optional opt-out — for most local service businesses, we lean toward allowing those too.

The table below lists each robots.txt token, what the vendor says it does, and what blocking it actually costs you. Every purpose comes from the vendor's own documentation, linked in the sources at the bottom.

Two entries deserve a closer look.

Google-Extended is not a pure training switch. Google's crawler documentation says it also governs grounding — "providing content from the Google Search index to the model at prompt time" — in Gemini Apps and Grounding with Google Search on Vertex AI. Blocking it can keep your pages out of Gemini app answers. It does not touch AI Overviews, which run on Googlebot.

Bingbot matters far beyond Bing. Bing Webmaster Tools now reports citations across Microsoft Copilot, Bing's AI summaries and partner integrations, and Cheers observed ChatGPT running Bing searches for local queries. Blocking Bingbot to "stop AI" would cost you more than any other single line.

Why do we lean toward allowing training crawlers? A local service business publishes marketing content: service areas, licenses, hours, what a repair involves. Having models learn that your company exists and what it does is rarely a loss. If you publish original research, detailed pricing guides or paid content, blocking training bots is a reasonable choice, and it won't cost you citations.

What should a local business robots.txt look like?

A good local business robots.txt explicitly allows the answer crawlers, keeps Googlebot, Bingbot and Applebot open through the default group, and, optionally, disallows training-only crawlers in a group of their own.

Here is a clean example for a WordPress site. Replace the domain in the Sitemap line. If you would rather allow training bots, delete the middle group entirely.

# Answer engines: allowed. These decide whether AI can cite you.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: DuckAssistBot
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Optional: training-only crawlers. Blocking these does NOT
# remove you from AI answers. Delete this group to allow them.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

# Everyone else, including Googlebot, Bingbot, Applebot
# and Google-Extended.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

A few choices in that file are deliberate:

• The answer-engine group repeats the admin rules. A crawler that matches its own group ignores the * group completely (more on that below), so any rule you want it to follow must be in its group.
• Google-Extended is left open. It stays in the default group, so your content remains available for grounding in Gemini Apps.
• Applebot stays open. Only Applebot-Extended, the training control, is in the optional block. Blocking Applebot itself would cost you Siri and Spotlight.
• Nothing blocks your service pages. Never disallow /services/, /locations/, or your FAQ and contact pages. Those are what answer engines quote.

Robots.txt only controls crawling. It isn't where you tell engines what your business does — that job belongs to your pages, your structured data and your Google Business Profile. Our guide on how to get recommended by ChatGPT covers the rest.

How do crawlers decide which robots.txt rule applies?

Under RFC 9309, a crawler uses the group that names it and ignores the * group, and inside that group the longest matching rule wins, with Allow winning a tie.

Most robots.txt mistakes come from assuming rules add up. They don't. In plain English, the standard works like this:

1. Find your group. The crawler looks for a User-agent line with its own name. Matching is case-insensitive, so gptbot and GPTBot are the same.
2. Specific beats star. If a group names the crawler, it follows that group only. The * group is used only when no group names it. If neither exists, nothing is blocked.
3. Duplicates merge. If the same crawler is named in two groups, the rules are combined into one.
4. Longest rule wins. Within the group, the rule with the most characters matching the URL applies. Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file.
5. Ties go to Allow. If an Allow and a Disallow match equally, the standard says Allow should be used.
6. No match means allowed. A URL that no rule covers can be crawled.

The classic accident looks like this: a site has User-agent: * with Disallow: / left over from a staging build, and the owner adds User-agent: Googlebot with Allow: / to fix Google. Googlebot is fine. OAI-SearchBot, PerplexityBot and Bingbot fall through to * and are blocked from the whole site.

The standard also covers what happens when robots.txt itself fails. If the file returns a 4xx error like 404, crawlers may access everything. If it returns a 5xx server error, crawlers must assume complete disallow. A robots.txt that intermittently throws 500 errors can quietly lock AI crawlers out.

Can a CDN or firewall block AI crawlers even when robots.txt allows them?

Yes — a CDN, firewall or security plugin can reject an AI crawler before it ever reads robots.txt, so a perfect file does nothing if the network layer says no.

Robots.txt is a polite request. A firewall is a locked door. When a bot-protection rule answers a crawler with a block or a challenge page, the crawler never gets your content, whatever robots.txt says.

Cloudflare, for example, has made AI crawler controls part of setup:

• Since July 1, 2025, every new domain is asked at sign-up whether to allow AI crawlers.
• On September 15, 2026, Cloudflare replaced its single "Block AI bots" switch with separate policies for Search, Training and Agent bots. It defines Agent as "user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents."
• Per Cloudflare's documentation, new domains now default to blocking Training and Agent bots on pages that display ads, with Search allowed. The legacy setting still lives under Security Settings.
• Cloudflare's optional managed robots.txt prepends Disallow rules for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent to your own file.

Note that last list includes Google-Extended. If you turned on the managed file to "stop training," you may also have opted out of Gemini grounding without knowing it.

What to check if you use Cloudflare or a similar service:

1. Open your security settings and find the AI bot policy. Make sure Search is allowed.
2. Decide deliberately on Agent. Blocking it can stop an assistant from opening your page when a customer asks it to.
3. Look at your live robots.txt in a browser. If your CDN is prepending rules, you will see them there, not in the file on your server.
4. Ask your host whether its bot protection or any security plugin blocks AI user agents. The same override can happen outside Cloudflare.

Do I need an llms.txt file?

No — llms.txt is an optional proposal, and no major answer engine has confirmed that it changes local recommendations; get robots.txt and your firewall right first.

The llms.txt proposal was published by Jeremy Howard on September 3, 2024, with a second version in August 2026. It suggests a Markdown file at your site root that gives AI agents a short map of your most useful pages.

It's cheap to add and harmless. But it's not an access control, and it's not a ranking signal anyone has confirmed. Google says in its AI features documentation that you "don't need to create new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode.

Our honest take: if your site has a clear structure, llms.txt is a five-minute housekeeping task, not a strategy. Our llms.txt generator builds one if you want it. Spend the real effort on the pages it would point to.

How do I check which AI crawlers can reach my site?

Run your domain through an AI crawler checker to test robots.txt against each bot, then confirm in your server logs that the crawlers are actually arriving and getting 200 responses.

Checking has two halves: what your rules say, and what really happens.

1. Test the rules. Our free AI Crawler Checker fetches your live robots.txt and applies RFC 9309 matching for each AI crawler, so you see exactly which bots are blocked and which line is responsible. It separates answer-engine blocks, which cost you citations, from training blocks, which don't.

2. Confirm in your logs. Robots.txt only shows intent. Your raw access logs show real visits. Download them from your hosting control panel and search for the crawler names:

grep -Ei "OAI-SearchBot|ChatGPT-User|GPTBot|PerplexityBot|Claude-SearchBot|ClaudeBot|bingbot|Googlebot" access.log | awk '{print $9}' | sort | uniq -c

That counts responses by status code. You want 200s. A pile of 403s means a firewall is blocking the bot. Repeated 503s or challenge pages usually mean bot protection or rate limiting is getting in the way.

3. Verify the bot is real. Anyone can put "GPTBot" in a user-agent string. OpenAI publishes IP lists for OAI-SearchBot, GPTBot and ChatGPT-User, Perplexity publishes one for PerplexityBot, and Anthropic says it publishes a list of source IPs.

For Google, the recommended check is a reverse DNS lookup: the hostname should end in googlebot.com, google.com or googleusercontent.com and resolve back to the same IP.

4. Watch the reports. Bing Webmaster Tools' AI Performance report shows when your pages are cited in Copilot and Bing's AI answers. If you use Cloudflare, its AI Crawl Control dashboard shows AI crawling patterns by bot.

Crawler access makes you eligible to be cited; it doesn't make you the answer. Once the gate is open, grade the pages that crawlers will read, then work through the full AEO checklist. If you want it handled for you, that's what our answer engine optimization service does.

Frequently asked questions

Should I block GPTBot?

Only if you do not want your content used to train OpenAI's models. OpenAI documents that GPTBot is its training crawler and that ChatGPT search uses a separate crawler, OAI-SearchBot. Blocking GPTBot does not remove you from ChatGPT search, but blocking OAI-SearchBot does.

What is OAI-SearchBot?

OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT's search features. If robots.txt blocks it, ChatGPT search cannot index or link your pages. OpenAI says robots.txt changes can take about 24 hours to take effect.

Does blocking Google-Extended remove me from AI Overviews?

No. Google states that Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode use normal Googlebot crawling. Google-Extended does control whether your content is used for Gemini training and for grounding answers in Gemini Apps and Vertex AI, which is why we recommend leaving it allowed.

Is PerplexityBot used to train AI models?

Perplexity says PerplexityBot is used to surface and link websites in Perplexity search results and is not used to train AI models. It respects robots.txt. Perplexity-User, which fetches pages when a user asks a question, generally ignores robots.txt because the request comes from a person.

How long does a robots.txt change take to work?

It depends on the crawler. RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, OpenAI says its systems take about 24 hours to adjust, and DuckDuckGo says a DuckAssistBot block takes effect after 72 hours. Check your server logs a few days after any change.

Will blocking AI crawlers protect my website from being copied?

Only partly. Robots.txt is a request that reputable crawlers honor, not an access control, and several vendors say their user-triggered fetchers may not follow it. If you need to keep content private, put it behind a login rather than relying on robots.txt.

Sources: OpenAI, Overview of OpenAI crawlers · Perplexity, Perplexity crawlers · Anthropic, Does Anthropic crawl data from the web · Google, Common crawlers (Google-Extended) · Google, AI features and your website · Google, Verifying Googlebot · Apple, About Applebot · Bing, AI Performance in Bing Webmaster Tools · DuckDuckGo, DuckAssistBot · Meta, Web crawlers · Amazon, Amazonbot · Common Crawl, CCBot · IETF, RFC 9309 Robots Exclusion Protocol · Cloudflare, Search, Training and Agent bot policies (Sept 15, 2026) · Cloudflare, Block AI bots · Cloudflare, Managed robots.txt · Cloudflare, July 1, 2025 press release · llmstxt.org · BrightLocal, Local Consumer Review Survey 2026 · Cheers, What sources does ChatGPT use