The short answer
Most businesses that want to be recommended should not block the crawlers that assistants use to look pages up and link to sources, because a page an assistant cannot fetch cannot be cited.
Whether to block crawlers that collect pages for model training is a separate business choice. Set each one deliberately in robots.txt.
Which crawlers are which
AI companies publish the names of their crawlers and what each is for. The list below reflects their documentation at the time of writing. Names and purposes do change, so check each vendor's own documentation before you rely on it.
| Name | Run by | What it is described as doing |
|---|---|---|
| GPTBot | OpenAI | Collects pages that may be used to train models |
| OAI-SearchBot | OpenAI | Finds pages to show and link to in ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a person in ChatGPT asks for it |
| Google-Extended | A token, not a crawler: controls use of content for Gemini models and grounding. Does not affect Google Search | |
| PerplexityBot | Perplexity | Finds pages to show and link to in Perplexity answers |
| ClaudeBot | Anthropic | Collects pages that may be used to train Claude models |
| CCBot | Common Crawl | Builds a public web archive that many AI projects have used for training |
Training and answering are different
The key distinction is between crawlers that collect pages for training and crawlers that fetch pages to answer a question now.
- Blocking a training crawler mostly affects future models.
- Blocking an answering crawler can stop an assistant from reading your current pages and linking to you.
- Blocking both by accident is common: many sites do it with one broad rule, or through a security setting at their host.
Robots.txt is a request, not a lock. Well-behaved bots follow it; some do not. It also cannot remove what a model already learned. Treat it as a way to state your preferences clearly, not as protection.
A sample robots.txt
This example lets search engines and answering crawlers in, opts out of training crawlers and keeps an admin area private. Adjust it to your own choices; there is no single right setup.
# Search engines: leave these alone
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Assistants fetching pages to answer and link to you
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Training crawlers: your choice (this example opts out)
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.co.uk/sitemap.xmlFive steps to set it up
- Open your current robots.txt. Visit yoursite.co.uk/robots.txt in a browser. Note any lines that mention AI crawlers, and any broad Disallow rules that might block everything.
- Decide which uses you are happy with. Separate two questions: may AI companies use your pages to train models, and may assistants fetch your pages to answer questions and link to you? You can answer them differently.
- Write one rule per bot. Add a User-agent line for each crawler you want to treat differently, followed by Allow or Disallow. Leave search engine crawlers such as Googlebot and Bingbot alone unless you mean to leave search.
- Check your firewall or CDN as well. Some hosting and security services block bots on their own. A robots.txt that allows a crawler does nothing if the firewall turns it away.
- Test and re-check. Fetch robots.txt again to confirm it is live, and review it every few months: vendors add and rename crawlers, so check each vendor's documentation.
Robots.txt only controls access. Whether a page is worth quoting is a separate matter: see how to write a page AI assistants can quote. You can also offer a short guide to your site with llms.txt, though it is only a proposal.
Honest limits
- What site audits check. GEO Agency's site audits check whether pages are reachable and readable, among other things.
- Estimates only. The tools run on Google Gemini, so results are estimates, not a log of which bots visited you or what any assistant said.
- No promises. We do not promise rankings or a place in AI answers.
Common questions
If I block GPTBot, will ChatGPT stop mentioning me?
Not necessarily. GPTBot is described by OpenAI as its crawler for training. ChatGPT search uses a different crawler, OAI-SearchBot, and ChatGPT-User fetches pages when a person asks. Blocking one does not block the others, and what a model already learned stays.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended is a robots.txt token, not a separate crawler. Google describes it as controlling whether content is used for Gemini models and grounding. Google Search crawling is done by Googlebot and is not affected.
Is robots.txt legally binding?
No. It is a long-standing convention. Reputable crawlers follow it; badly behaved ones may not. For content that must stay private, use a login, not robots.txt.
What is a sensible default for a business that wants customers?
For most businesses that want to be recommended: allow search engines, allow the crawlers assistants use to look things up and link to sources, and decide separately on training crawlers. There is no right answer on training; it is a business choice.