What robots.txt tells named AI crawlers
The companies behind AI tools, and the Common Crawl archive, publish the names robots.txt can address and say what each one covers: training, search results, a page a person asks for, using pages for AI training and grounding, or an open web archive. When your robots.txt has a group for one of those names, the full report says what the file asks of it, in the Site profile. It is a fact about a choice you made, not a problem to fix, and it changes no score.
Updated · By Martin Clavell
What we detect
The robots.txt your production storefront serves at its root — the file every audit already reads. For each name below that a User-agent: group addresses, the report says whether the file asks it to stay out of the whole store, out of some pages, or out of none, and points at the group that says so.
- Names are matched exactly, ignoring case and any version number written after the name. A group for a different name is never read as one of these.
- A file whose only group is
User-agent: *names none of these crawlers, so it gets no row. - When robots.txt does not answer with a normal page, the check is reported as not checked — never as allowed.
- A sandbox is not checked: its robots.txt usually turns every crawler away on purpose.
The names we report
Each name is described the way its operator’s own page describes it. Operators rename, add and retire these names without notice, so the list follows their pages.
GPTBot: OpenAI’s crawler that collects pages for model trainingOAI-SearchBot: OpenAI’s crawler for search results in ChatGPTChatGPT-User: OpenAI’s agent for actions people take in ChatGPTClaudeBot: Anthropic’s crawler that collects pages for model trainingClaude-SearchBot: Anthropic’s crawler for search resultsClaude-User: Anthropic’s agent for questions people ask ClaudeGoogle-Extended: Google’s Gemini training and groundingApplebot-Extended: Apple’s foundation model trainingPerplexityBot: Perplexity’s crawler for its search resultsPerplexity-User: Perplexity’s agent for actions people take in PerplexityCCBot: Common Crawl’s crawler for its open web archive
What this does not tell you
- Whether any AI product shows, ranks or quotes your store, or sends it visitors. That happens inside another company’s systems, which no audit can see from outside, so the report never claims it either way.
- What a crawler actually does. The report says what your file asks. For an agent that fetches a page when a person asks it to, OpenAI and Perplexity say robots.txt rules may not apply.
- Whether asking a training crawler to stay out affects a company’s search features. Each name covers only what its operator says it does, and the operators draw that line differently, so the report says nothing about one name from another.
- Anything about Google Search. Google says its Gemini control does not affect a site’s inclusion in Google Search and is not used as a ranking signal there.
If you want to change it
Nothing here needs fixing: whether each crawler may read your store is your decision. If the file says something you did not intend:
- Edit robots.txt: add a
User-agent:group for the name withDisallow: /to ask it to stay out of the whole store, or remove its group to leave it to your*rules. - Upload the file to the hosting root. It replaces the robots.txt NetSuite generates, so keep any rules you rely on.
- Re-run the audit and read the Site profile in the full report.
In NetSuite: Commerce > Hosting > Website Hosting Files, the robots.txt file in the site's hosting root folder Oracle documentation
Questions
- Does turning an AI crawler away lower my score?
- No. The row is a fact in the Site profile and scores nothing, whichever way your file answers.
- Will letting these crawlers in put my store in AI answers?
- Nobody outside the companies that run them can tell you that, and the report does not try. It states what your file asks; what each company does with your pages is theirs to describe.