Skip to content

robots.txt checker for AI crawlers, with llms.txt

Can ChatGPT, Claude and other AI assistants read your website? Enter your domain: Listentic checks your robots.txt for 14 AI and search crawlers, looks for an llms.txt and a sitemap, and drafts an llms.txt for you.

What the check covers

  • robots.txt rules for 14 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta and Common Crawl.
  • Whether an llms.txt exists (an HTML page in its place doesn't count).
  • Whether a sitemap is linked or available.
  • Whether your homepage is set to noindex.
  • Whether your site runs on Cloudflare, which can block AI crawlers separately.

Search crawlers and training crawlers

AI companies use different crawlers. OAI-SearchBot, Claude-SearchBot and PerplexityBot load pages to answer questions and cite sources; ChatGPT-User, Claude-User and Perplexity-User load a page when someone asks about it. GPTBot, ClaudeBot and Google-Extended collect training data. You can block training and still allow search: then AI assistants can cite you without using your content for training.

What is llms.txt?

llms.txt is a short Markdown file at the root of your website that tells AI assistants what your site is about and which pages matter most. It is a proposal, not an official standard, and not every AI provider reads it yet. Adding one takes a few minutes, and the draft from the check gives you a starting point.

How we check

The check runs as ListenticBot and only loads your robots.txt, llms.txt, sitemap and, if your robots.txt allows it, your homepage. It doesn't pose as another crawler, and the results aren't stored.

Questions and answers

Does blocking GPTBot keep me out of ChatGPT?

Not necessarily. GPTBot collects training data, while ChatGPT search uses OAI-SearchBot and ChatGPT-User. The result shows both groups separately.

Do I need an llms.txt?

It's optional. It helps AI assistants understand your site faster, but whether they mention you depends mostly on whether your pages can be crawled and on what other websites say about you.

How often can I use the check?

Three times a day without an account. With a Listentic account you get more checks, and the monthly website check of your projects covers AI crawlers automatically.

Is the check free?

Yes, completely free and without sign-up.