Menu

Choose how AI assistants use your site

At the end of this guide your site tells AI crawlers (the programs AI companies use to read websites) what they may do with your pages.

The three choices

The three choices
ChoiceWhat the dashboard says
AllowAI crawlers may read the site, answer questions from it and train on it.
No trainingAI crawlers may read the site to answer questions, but are asked not to train on it.
BlockAI crawlers are asked not to read the site at all.

Allow is the default. Search engines such as Google keep listing your site whatever you choose.

Change the setting

  1. In your site's dashboard, open Settings, then Integrations.
  2. Under AI crawlers, choose Allow, No training or Block.
  3. Choose Save changes.
  4. Publish Site settings: on the Overview, choose Publish, select Site settings and publish it. See Publish your changes.
Settings: aI crawlers, Analytics, consent, Search Console.

The setting "takes effect when you next publish the settings". Until then the live site keeps the previous choice.

What each choice writes

What each choice writes
AllowNo trainingBlock
robots.txtLets every crawler read the whole siteAsks the training crawlers listed below to stay away, and says the site may be used for search and answers but not for trainingAsks the training and the answer crawlers listed below to stay away, and says the site may be used for search only
llms.txt and the .md page copiesWrittenWrittenNot written
Business details and page descriptions for search enginesWrittenWrittenWritten

The "says" part is a Content-Signal line in robots.txt, a newer way for a site to state how its content may be used.

Which crawlers

Which crawlers
KindCrawlers named in robots.txt
Training crawlers (asked to stay away with No training and Block)GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent
Answer crawlers (also asked to stay away with Block)OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User

Google-Extended only controls whether Google uses your pages for its AI models; it does not affect Google Search.

What this cannot do

The dashboard puts it plainly: "This is a request crawlers are expected to honour." The AI companies publish these crawler names so that sites can make this request, but a crawler that ignores robots.txt can still read a public page. Anything published on your site is public; to keep something private, do not publish it.