Iris · AI SEO, AEO & GEO auditor

← Back to Iris

Free tool · no sign-up

AI crawler checker: can ChatGPT, Claude and Perplexity read your site?

Type a website. Iris reads its robots.txt and shows, crawler by crawler, who is let in: the eight that feed AI search, the three that open a page when a person asks, and the eight that collect training data.

Free, no sign-up. Iris reads one file, robots.txt, as IrisBot. It never pretends to be another crawler.

What does the AI crawler checker look at?

One file: /robots.txt on the site you type. For each of 19 crawler names it finds the group of rules that crawler obeys and gives one of four answers.

  • Allowed: a group names the crawler and does not close the site to it.
  • Allowed, no rule: nothing names the crawler, so it follows the rules for everyone, and those leave the site open.
  • Partly blocked: the crawler has its own group that closes part of the site. The lines are shown.
  • Blocked: the crawler is told to stay out of the whole site, by its own group or by the rules for everyone.

Which crawlers decide whether an AI assistant can use your pages?

The ones that build a search index. They are different crawlers from the ones that collect training data, and the difference is the most common mistake in robots.txt files.

  • ChatGPT search uses OAI-SearchBot. GPTBot collects training data.
  • Claude uses Claude-SearchBot for search. ClaudeBot collects training data.
  • Perplexity uses PerplexityBot.
  • Google AI Overviews and AI Mode use the same crawl as Google Search, Googlebot. Google-Extended is a separate switch for Gemini training and has no effect on Search.
  • Microsoft Copilot has no crawler of its own. It draws on Bing, so the name that matters is Bingbot.

Does blocking GPTBot take my site out of ChatGPT?

No. GPTBot is the training crawler. ChatGPT’s search answers come from OAI-SearchBot, so a site can keep its pages out of model training and still be found and cited. The same holds for ClaudeBot and Claude-SearchBot, and for Google-Extended and Googlebot. The checker lists the two groups separately so that one decision does not make the other by accident.

What can robots.txt not tell you?

robots.txt is a request, not a firewall, and this check reads only that file.

  • A rule at your CDN or host can block a crawler with no line in robots.txt. Some CDNs have a single switch that blocks AI crawlers.
  • Visits made because a person asked an assistant to open a page do not always follow the file. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.
  • Being allowed is not being cited. It only means the assistant may read the page.

The full Iris audit opens the page itself and tells you if our reader was blocked on the way in, then scores what a crawler finds there.

Questions people ask about this

Is the AI crawler checker free?
Yes. There is no sign-up and nothing to pay. One connection can run 40 checks a day.
What does "Allowed, no rule" mean?
No group in the file names that crawler, so it follows the rules written for all crawlers, and those do not close the site. In practice the crawler is allowed.
Should I block AI training crawlers?
That is your decision. Blocking them keeps your pages out of model training and does not change AI search, which uses different crawlers. The checker shows the two groups separately for that reason.
Why did the checker say it could not read robots.txt?
The server refused our reader, answered with an error or did not answer in time. That is not a verdict about the site: other crawlers may be served normally. Open the file in a browser to see what it says.
Does Iris pretend to be GPTBot or another crawler when it checks?
No. It reads robots.txt as IrisBot and reports what the rules say for each crawler name. It does not send requests under another crawler's name.

The other free tools

Crawler names and what each is for were read from the vendors’ own pages on 10 November 2026: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and Common Crawl. Vendors add crawlers; if a name is missing, tell us.