Is your website blocking AI engines?
One line in robots.txt can remove you from ChatGPT, Perplexity or Copilot answers. Check 18 AI crawlers in seconds and see exactly which rule does what.
Free · No email needed · We only read your public robots.txt and llms.txt.
We’ll email your report link and what to fix first, then re-run this check every month and tell you what changed. Stop anytime with one click.
How does the AI Crawler Checker work?
We fetch robots.txt
We read the public robots.txt file at the root of your domain, the same file every crawler checks first.
We apply the real rules
Each of 18 AI crawlers is matched to its user-agent group using the official robots.txt standard (RFC 9309), longest rule wins.
We separate what matters
Answer-engine crawlers decide whether you can appear in AI answers. Training crawlers only decide whether your content trains models.
You get the exact fix
For every blocked answer engine you see the rule responsible and what to change.
AI Crawler Checker FAQ
Should I block AI crawlers?
Block training crawlers like GPTBot, ClaudeBot or CCBot if you don't want your content used to train models — it won't remove you from live answers. Don't block answer-engine crawlers such as OAI-SearchBot, PerplexityBot, Bingbot or Googlebot, or you can't be cited.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether your content is used for Gemini models. AI Overviews and AI Mode use normal Googlebot crawling, so blocking Googlebot is what would remove you.
My robots.txt allows everything — why am I still not in ChatGPT?
Crawler access only makes you eligible. Some hosts and firewalls also block AI bots before robots.txt is even read. Beyond access, engines need consistent business facts, reviews and clear pages to recommend you. Try the AI Visibility Checker next.
What is llms.txt?
A proposed file that gives AI systems a plain-text map of your site. It's optional, and no major engine has confirmed it changes local recommendations. We show whether you have one for completeness.
Keep going
SEO Audit Tool
Full SEO + AI-readiness audit with a fix list.
Open →AI Visibility Checker
See if ChatGPT, Gemini, Claude and more recommend you.
Open →Answer-Readiness Grader
Score any page for AEO, with a fix list.
Open →Schema Generator
LocalBusiness + FAQ JSON-LD in a minute.
Open →llms.txt Generator
Build an llms.txt file for your site.
Open →Want us to do this for you?
We run answer engine optimization for local service businesses: we fix the sources AI engines read, publish answer-ready content and track your share of AI answers every month. Month-to-month, and the co-founder answers the phone.