Part of the SiteMap Series
In the previous lesson, you looked at how sitemaps shape crawl budget allocation at scale, treating the search engine crawlers as a single category. AI crawlers complicate that picture. They share infrastructure patterns with traditional crawlers but operate with different objectives, different priorities, and different relationships to the sitemap.
This matters because the AI Search era has multiplied the number of automated agents reading your sitemap. Where five years ago “the crawler” effectively meant Googlebot and Bingbot, today’s sitemap is also read by GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bytespider, CCBot, and a long list of others. Each has its own purpose, its own crawl pattern, and its own use (or non-use) of sitemap data. Setting up sitemaps for the AI Search era means understanding what these crawlers actually do with the file.
The honest framing: AI crawler behaviour is less documented than search engine crawler behaviour. The major AI companies publish less about their crawl behaviour than Google publishes about Googlebot. This lesson covers what is known from official documentation, what is observable from server logs, and where the uncertainties sit. The goal is to set up sitemaps that work well for both search and AI crawlers without making speculative changes based on assumed behaviour.
How AI crawlers differ from traditional search crawlers
Traditional search crawlers and AI crawlers share the basic mechanics of fetching pages and processing content. The differences sit in their objectives.
Search crawlers prioritise indexing for retrieval. Googlebot, Bingbot, and similar crawlers exist to build an index that powers search results. They prioritise comprehensive coverage of canonical URLs, frequent re-crawling of fresh content, and signals that help rank pages against queries. The sitemap is a strong signal because it tells the crawler which URLs the publisher wants in the index.
AI training crawlers prioritise broad coverage of high-quality content. Crawlers like GPTBot, ClaudeBot, and CCBot exist to gather training data for language models. They prioritise content diversity, content quality, and crawl efficiency. The sitemap is useful for URL discovery, but their priorities differ from search: they want broad sampling rather than maximum freshness, and they care less about ranking signals.
AI search crawlers fall between the two. PerplexityBot and OAI-SearchBot crawl for AI-powered search products. Their behaviour resembles traditional search crawlers more closely (URL discovery, re-crawl prioritisation, freshness signals), because they need current information to answer queries.
On-demand browsing agents work differently still. ChatGPT-User and equivalent agents from other AI companies fetch specific URLs when triggered by user actions (a user pastes a URL, asks the model to read a page, or requests browsing for a query). These agents typically do not crawl entire sites or read sitemaps; they fetch individual URLs on demand.
The practical effect: a single sitemap is read by crawlers with different objectives. The same lastmod, the same URL list, the same structure gets interpreted differently depending on which crawler is fetching it.
The major AI crawlers operating today
The AI crawler space is fragmented. Different companies operate different crawlers for different purposes. Knowing which crawler does what helps you set up sitemap access (and robots.txt rules) intentionally.
Training-focused crawlers:
- GPTBot (OpenAI). Crawls for OpenAI model training. Respects robots.txt. Uses sitemaps for URL discovery.
- ClaudeBot (Anthropic). Crawls for Anthropic model training. Respects robots.txt. Sitemap usage less documented than GPTBot.
- Bytespider (ByteDance). Crawls for ByteDance AI products including TikTok. Has been observed crawling aggressively; many sites block it.
- CCBot (Common Crawl). The foundation crawler whose data is used by many AI companies as a training source. Predates the AI era but has become central to it.
Search-focused AI crawlers:
- OAI-SearchBot (OpenAI). Crawls for ChatGPT’s web search feature. Separate from GPTBot. Behaves more like a traditional search crawler.
- PerplexityBot (Perplexity AI). Crawls for the Perplexity search product. Uses sitemaps actively for URL discovery and re-crawl prioritisation.
On-demand browsing agents:
- ChatGPT-User (OpenAI). Fetches URLs when ChatGPT users browse or paste URLs.
- Equivalent agents from Anthropic and Perplexity. Anthropic and Perplexity operate similar on-demand fetching mechanisms for user-triggered browsing; user-agent strings have evolved over time, so check current documentation rather than relying on remembered values.
Opt-out tokens (not crawlers):
- Google-Extended. A robots.txt token that lets publishers opt out of Google’s AI products (Gemini, Vertex AI) using their content for training. Does not crawl independently; uses Googlebot’s existing data.
- Applebot-Extended. Apple’s equivalent for opting out of their AI training (Apple Intelligence). Operates similarly.
- Meta-ExternalAgent and Meta-ExternalFetcher. Meta’s AI crawlers, with similar control mechanisms.
The list is not exhaustive and evolves. New crawlers appear; existing ones change behaviour. The point is recognising the categories: training crawlers, search crawlers, browsing agents, opt-out tokens. Your sitemap interacts with each category differently.
What AI crawlers actually do with your sitemap
Three observations from public documentation and observable crawl behaviour.
They use the sitemap for URL discovery. Every documented AI crawler that fetches sitemaps uses them primarily for URL discovery. The sitemap tells the crawler what URLs exist; the crawler then decides which to fetch based on its own priorities. This use is consistent across training crawlers and search-focused AI crawlers.
They likely use lastmod for prioritisation, but the weight varies. Search-focused AI crawlers (PerplexityBot, OAI-SearchBot) appear to use lastmod similarly to traditional search crawlers, prioritising recently updated content. The trust-building dynamic covered in Lesson 6 of this module: How Sitemaps Affect Your Crawl Budget at Scale applies to these crawlers as much as it does to Googlebot. Training crawlers may use lastmod less aggressively because their goal is broad coverage rather than current information; a stable canonical article is as valuable to training as a freshly updated one.
They ignore most sitemap metadata beyond loc and lastmod. The priority and changefreq fields are largely ignored, the same as with traditional search crawlers. Image and video extensions are used only by crawlers that specifically index multimedia content. Hreflang annotations may help AI crawlers detect language and regional variants, but the practical effect is unclear.
What this means in practice: the sitemap remains a useful signal for AI crawlers, but the parts of it that matter most are the basics. A clean, accurate URL list with truthful lastmod values serves AI crawlers well. The fancier sitemap features have less effect than they do with traditional search engines (where their effect is already limited).
Controlling AI crawler access while keeping discovery clean
Many publishers want to control which AI crawlers can access their content without breaking traditional search indexing. The sitemap itself does not control access; access control sits in robots.txt and HTTP-level rules. But the two interact.
Three access control patterns to know.
1. Block specific AI crawlers entirely
Add disallow rules in robots.txt for specific user agents. Example:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /This blocks the crawler from fetching any URL on the site, including the sitemap. If you block the crawler this way, sitemap content is irrelevant for that crawler.
2. Allow crawling but opt out of AI training
Use opt-out tokens like Google-Extended and Applebot-Extended. Example:
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /This lets Googlebot and Applebot crawl normally for search but prevents your content from being used to train the AI products. The sitemap continues to serve traditional search while AI training is excluded.
3. Allow everything
No robots.txt restrictions on AI crawlers; all crawlers (traditional and AI) can access the sitemap and all URLs. This is the default and remains the right choice for many publishers because AI search traffic is becoming a meaningful traffic source.
The interaction with the sitemap: the sitemap should always be allowed in robots.txt (allow all user agents to fetch sitemap.xml), even when individual content URLs are restricted. Blocking sitemap access entirely does not improve privacy or security; it just makes discovery harder for the crawlers you do want to reach.
Where llms.txt fits, and where it does not
The proposed llms.txt standard surfaces in AI Search conversations frequently. The honest framing matters because the standard is often discussed as if it were established.
What llms.txt is. A proposed standard for a markdown file at the site root (similar location to robots.txt) that tells AI systems which content on the site is most valuable for them to understand. It is structured as a curated content map, not a crawl directive. The proposal comes from Jeremy Howard and Answer.AI; it was published in 2024 and has gained some adoption.
What llms.txt is not. It is not currently supported as an authoritative signal by any major AI company (OpenAI, Anthropic, Google, Perplexity). No published documentation from these companies references llms.txt as a file they read. The adoption is publisher-side; the consumption side has not caught up.
How it relates to your sitemap. llms.txt is not a replacement for the sitemap. The sitemap remains the canonical URL discovery mechanism for AI crawlers. llms.txt, if you choose to publish one, sits alongside the sitemap and provides additional context: which URLs matter most, what topics they cover, how the content is organised conceptually.
Should you publish one? It is a reasonable optional artefact for sites that want to participate in the emerging convention. The cost is low (a markdown file at the root). The benefit is speculative because the major AI companies have not committed to reading it. Treat llms.txt as a low-priority addition rather than a required signal.
The sitemap remains the load-bearing artefact for AI crawler URL discovery. llms.txt is an experimental supplement, not a replacement.
What this means for your sitemap setup
The practical implications for setting up sitemaps in the AI Search era are smaller than the conversation around them often suggests.
Keep the same fundamentals. A clean, accurate sitemap with truthful lastmod values, sensible structure, and exclusion of non-canonical URLs serves AI crawlers as well as it serves search engines. The patterns from Modules 1-3 and the earlier lessons of Module 4 still apply.
Make sure your sitemap is accessible to the crawlers you want to reach. Check your robots.txt rules. If you have blocked specific AI crawlers, you have effectively closed off your sitemap to them too. That may be the intentional choice; just be deliberate about it.
Consider whether to allow AI crawlers at all. This is a strategic question rather than a technical one. AI search referrals are becoming a meaningful traffic source. Blocking AI crawlers entirely cuts off both AI training (which most publishers care less about) and AI search visibility (which most publishers care more about). The opt-out tokens (Google-Extended, Applebot-Extended) offer a middle path: allow search crawling, opt out of training.
Add llms.txt if you want, but do not overinvest. A simple llms.txt at the root is a low-cost optional artefact. Do not rebuild your content strategy around it.
Monitor AI crawler behaviour through server logs. Search Console covers Googlebot crawl stats. To see what AI crawlers are doing, you need server log analysis. Tools like Screaming Frog Log File Analyser, Botify, or Lumar let you filter by user agent and see crawl patterns for GPTBot, ClaudeBot, PerplexityBot, and the rest.
The headline: AI crawlers do not change what makes a good sitemap. They expand the audience for the same sitemap. Set up well for search, monitor for AI crawler activity, and adjust access policies based on your strategic position rather than technical compliance.
Common gotchas to avoid
Four issues come up regularly with AI crawler and sitemap interactions.
1. Blocking AI crawlers but expecting AI search visibility
A site that blocks GPTBot in robots.txt should not expect to appear in ChatGPT search results. The same applies to ClaudeBot and Anthropic’s products, PerplexityBot and Perplexity search results, and so on. If AI search visibility is part of your strategy, your robots.txt needs to allow the relevant crawlers. The fix is to align your robots.txt rules with your strategic goals rather than copying blocking templates from other sites without thinking through the implications.
2. Confusing opt-out tokens with crawler blocks
Google-Extended is not a crawler; it is an opt-out token. Blocking Google-Extended in robots.txt does not block Googlebot from crawling; it tells Google not to use the crawled content for AI training. The same applies to Applebot-Extended. Confusing these with crawler blocks leads to either over-blocking (thinking you have stopped crawling when you have only opted out of training) or under-blocking (thinking you have opted out of training when you have only blocked the search crawler).
3. Treating llms.txt as established
llms.txt comes up in AI Search conversations as if major AI companies support it. They do not, at least not as of 2026. Implementing llms.txt is fine as an optional experiment; treating it as a required signal is a misallocation of effort. The sitemap remains the authoritative URL discovery mechanism.
4. Ignoring AI crawler traffic in monitoring
Server logs show AI crawler activity that Search Console does not. Publishers who only monitor Search Console miss a growing portion of their crawl activity. The fix is to add log analysis to the regular monitoring cadence and pay attention to which AI crawlers are fetching content, at what rate, and from which IP ranges.
Where this leaves us
You can now set up sitemaps with the AI crawler audience in mind: understanding which AI crawlers are reading your sitemap, what they extract from it, how your access policies interact with sitemap availability, and where experimental standards like llms.txt sit relative to the established signal of the sitemap itself.
The final lesson in this module brings the entire series together. After thirty-six lessons covering what sitemaps are, how to build them, how to specialise them, and how to apply them across real-world site types, the question becomes: how do you audit an existing sitemap setup? The audit framework synthesises every signal you have learned to evaluate so far, organised into a sequence that works for any site, any scale, and any architecture. That is the proper close of the Sitemap Series.
Up next: How to Audit Your Website Sitemap →
This is Module 4: Lesson 7 of The Sitemap Series, a Technical SEO series on sitemaps from first principles, built for the AI Search era.