How AI Crawlers Read Your Site

Before any AI system can cite your page, something has to fetch it. That fetching is done by crawlers, several companies run their own, and each one obeys a separate instruction in a file most site owners have never opened. This page explains who is visiting, how to see them, and how to decide what to allow.

Who is actually knocking

A crawler is an automated program that downloads pages the way a browser would, but with no person watching. It identifies itself with a user agent — a short name in the request, like Googlebot — and well-behaved ones check robots.txt first. That file lives at the root of your site, at yoursite.com/robots.txt, and lists which paths each named agent may fetch.

The important and frequently missed point is that these are separate permissions. Allowing Google says nothing about anyone else.

Roughly, the visitors fall into three groups:

• Search crawlers. Googlebot is the one that matters most. It feeds Google Search, and because AI Overviews are a feature of Search, the same crawl is what makes you eligible to appear in them. Block Googlebot and you leave both at once.
• Assistant crawlers. GPTBot, ClaudeBot, PerplexityBot and others, run by the companies behind those products. Each is controlled by its own name in robots.txt.
• Grounding and training controls. Google-Extended is not a crawler at all — nothing fetches under that name. It is a token you can disallow to say that your content should not be used for Google's generative products such as Gemini. Disallowing it does not remove you from Search and does not remove you from AI Overviews.

How to see them for yourself

You do not have to take a list on trust. Your server writes a line for every request it serves, and those lines carry the user agent. Search your access log for a name and you will see whether it has been by, how often, and what it asked for.

This is worth doing before changing anything, for a plain reason: it turns the question from "what should I allow in principle" into "who is actually reading my site". The answer is often surprising in both directions.

For Googlebot specifically, Search Console reports crawl activity directly, which is more reliable than reading logs by hand.

What to do

• Open yoursite.com/robots.txt and read it. Many sites have rules nobody remembers adding, copied from a template years ago.
• Confirm Googlebot is not blocked from anything you want found. This is the single highest-cost mistake available in the file.
• Decide the assistant crawlers deliberately, one at a time. Allowing them is how you become eligible to be cited; disallowing them is a legitimate choice about your work being used. Either is defensible. Doing it by accident is not.
• If you want to be found but not quoted at length, use the snippet controls rather than a block: nosnippet, max-snippet, and the data-nosnippet attribute on a specific element. These limit what can be shown, including in AI features, while leaving the page indexed.
• Re-check after any site migration. A Disallow: / left over from a staging server is the classic way to disappear.

Common mistakes

The most damaging is confusing robots.txt with a way to keep a page out of search results. It controls fetching, not indexing. A blocked page can still be listed if other sites link to it, and because the blocked page cannot be read, Google cannot see a noindex instruction on it either. To keep a page out of results, allow the crawl and use noindex.

The second is assuming Google-Extended affects Search. It does not. People disallow it expecting to opt out of AI Overviews and are then puzzled to still appear there, because AI Overviews follow Googlebot.

The third is blocking a crawler to "save bandwidth" without measuring anything. Check the logs first; on most small sites the traffic is trivial.

Sources: Google Search Central on Google crawlers and fetchers and AI features in Google Search. For the difference between blocking and de-indexing, see sitemaps and robots.txt.