Choose how AI assistants use your site
At the end of this guide your site tells AI crawlers (the programs AI companies use to read websites) what they may do with your pages.
| Choice | What the dashboard says |
|---|---|
| Allow | AI crawlers may read the site, answer questions from it and train on it. |
| No training | AI crawlers may read the site to answer questions, but are asked not to train on it. |
| Block | AI crawlers are asked not to read the site at all. |
Allow is the default. Search engines such as Google keep listing your site whatever you choose.
- In your site's dashboard, open Settings, then Integrations.
- Under AI crawlers, choose Allow, No training or Block.
- Choose Save changes.
- Publish Site settings: on the Overview, choose Publish, select Site settings and publish it. See Publish your changes.

The setting "takes effect when you next publish the settings". Until then the live site keeps the previous choice.
| Allow | No training | Block | |
|---|---|---|---|
| robots.txt | Lets every crawler read the whole site | Asks the training crawlers listed below to stay away, and says the site may be used for search and answers but not for training | Asks the training and the answer crawlers listed below to stay away, and says the site may be used for search only |
| llms.txt and the .md page copies | Written | Written | Not written |
| Business details and page descriptions for search engines | Written | Written | Written |
The "says" part is a Content-Signal line in robots.txt, a newer way for a site to state how its content may be used.
| Kind | Crawlers named in robots.txt |
|---|---|
| Training crawlers (asked to stay away with No training and Block) | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent |
| Answer crawlers (also asked to stay away with Block) | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User |
Google-Extended only controls whether Google uses your pages for its AI models; it does not affect Google Search.
The dashboard puts it plainly: "This is a request crawlers are expected to honour." The AI companies publish these crawler names so that sites can make this request, but a crawler that ignores robots.txt can still read a public page. Anything published on your site is public; to keep something private, do not publish it.
