How to Control AI Crawlers With robots.txt and llms.txt in 2026
GPTBot, ClaudeBot, and Google-Extended are hitting your site right now. Here's how to decide who gets in, block the ones you don't want with robots.txt, and whether the new llms.txt file is worth your time.

Open your server logs on any given day and you'll see names you didn't used to see. GPTBot. ClaudeBot. PerplexityBot. Google-Extended. CCBot. They're crawling your pages to train and ground large language models, and most site owners have no idea whether to welcome them or slam the door.
The decision isn't obvious. Block everything and you might vanish from the AI answers that are quietly eating a slice of search traffic. Allow everything and you're handing your content to models that may never send a visitor back. There's a middle path, and it starts with knowing exactly who's knocking.
Quick Answer: AI crawlers are bots that fetch your content to train or ground large language models, and you control them the same way you control search crawlers, through your robots.txt file. You add User-agent blocks naming each bot, then Disallow or Allow paths for it. Blocking works only for well-behaved crawlers that respect the rules. The newer llms.txt file is a proposed convention for pointing AI tools to your best content, but adoption is unconfirmed, so treat it as optional and low-cost, not a ranking lever.
- AI crawlers use named user agents you can target individually in robots.txt
- Blocking only works for bots that voluntarily honor robots.txt
- Google-Extended controls AI training and grounding, not Google Search indexing
- Blocking training bots does not remove you from Google's regular search results
- Allowing AI crawlers can earn citations in AI answers that link back to you
- llms.txt is an unofficial proposal, useful to try but not a guaranteed signal
- Decide per bot based on whether it sends traffic or only takes content
Which AI Crawlers Are Actually Hitting Your Site?
More than you'd guess, and they don't all do the same job. Some fetch pages to train future models, some fetch pages live to answer a user's question right now, and some do both. That distinction matters because a bot that answers questions live can cite you and send a click, while a pure training bot usually can't.
The big ones show up under distinct user-agent strings, which is what makes them controllable. Here's the current cast of characters worth knowing.
| Crawler | Who runs it | What it's for |
|---|---|---|
| GPTBot | OpenAI | Training data for future models |
| OAI-SearchBot | OpenAI | Live fetching for ChatGPT search results |
| ClaudeBot | Anthropic | Training and model data collection |
| Google-Extended | AI training and grounding opt-out token | |
| PerplexityBot | Perplexity | Indexing for its answer engine |
| CCBot | Common Crawl | Open dataset many models train on |
| Bytespider | ByteDance | Training data collection |
| Applebot-Extended | Apple | AI training opt-out token |
A few of these behave differently than their name suggests. Google-Extended isn't a crawler at all in the usual sense, it's a token you use in robots.txt to opt out of having your content used for Gemini training and grounding, and it does not affect how Googlebot crawls or ranks you for regular search. OAI-SearchBot and PerplexityBot fetch content to power answers users see immediately, so they're the ones most likely to cite and link you.
Watching your own logs is the honest way to know who's spending your crawl budget. If you run log file analysis, filter by user agent and you'll see the real volume each bot is pulling, which is far more useful than guessing from a list.
How Do You Block or Allow AI Crawlers in robots.txt?
You name the bot's user agent, then tell it where it can and can't go. It's the same syntax you already use for search engines, applied to a new set of names. Each bot reads the block addressed to it and, if it's a well-behaved crawler, obeys.
The structure is a User-agent line naming the bot, followed by Disallow or Allow rules. To block OpenAI's training crawler from your whole site, you'd write a block for GPTBot with Disallow: /. To let a live-answer bot in everywhere, you'd give it Disallow: with nothing after it, which permits the entire site. You can mix these freely, blocking training bots while welcoming the ones that cite you.
Here's what a selective setup looks like in plain terms. You block the pure training crawlers like GPTBot, ClaudeBot, CCBot, and Bytespider with a full Disallow: /, you add the Google-Extended and Applebot-Extended tokens the same way to opt out of their AI training, and you leave OAI-SearchBot and PerplexityBot allowed so your content can still appear and link in their answers. That combination keeps your work out of raw training sets while staying eligible for the AI results that actually drive visits.
Two cautions save you real pain here. First, robots.txt is voluntary, so it only stops crawlers that choose to respect it, and not every AI scraper does. Blocking a bot in robots.txt is a request, not a wall, and enforcement against bad actors needs server-level or firewall rules instead. Second, order and specificity matter, so test your file rather than assuming, since one misplaced Disallow: / addressed to * can wall off every crawler including Googlebot.
Does Blocking AI Crawlers Hurt Your Google Rankings?
No, and this is the fear that stops people from making any decision at all. Blocking AI training crawlers does not remove you from Google's search results, because the bot that indexes you for Search is Googlebot, and it's separate from the AI-training token.
Google draws a deliberate line here. Googlebot crawls and indexes your site for ranking in the results pages, while Google-Extended is a distinct token that only governs whether your content feeds Gemini's training and grounding. Disallowing Google-Extended tells Google not to use your pages for its generative AI products, and your organic rankings, your indexing, and your presence in the results stay exactly as they were. You can opt out of AI training and keep every bit of your search visibility.
The trade-off lives elsewhere. If you block the live-answer bots too, you may reduce your odds of being cited inside AI answers and assistants, which is a growing surface for discovery. This is where answer engine optimization comes in, since being present and citable in tools like ChatGPT and Perplexity increasingly matters alongside classic rankings. Blocking training and blocking live citation are two different choices with two different costs, and it pays to make them separately rather than reaching for one blanket rule.
For most publishers the sensible split is to opt out of raw training while staying open to the bots that fetch, cite, and link. You keep your search rankings untouched, you stop feeding models that give nothing back, and you preserve your shot at the AI citations that do send readers your way. If you want to know whether any of this is working, tracking your AI search visibility tells you whether the open bots are actually surfacing you.
Is llms.txt Worth Adding to Your Site?
It's cheap to add and unproven to matter, which is a fair summary of where things stand. The llms.txt file is a proposed convention, not an official standard, and the honest position is to treat it as a low-cost experiment rather than a ranking lever.
The idea is straightforward. You place a markdown file at /llms.txt that points AI tools to your most important pages and gives a clean, structured overview of your site, so a model reading it can find your best content without wading through navigation and boilerplate. It's meant to be for AI what an XML sitemap is for search crawlers, a curated map of what matters. The intent is genuinely useful, especially for documentation-heavy or content-heavy sites.
The catch is adoption. No major AI company has confirmed that its crawlers read or act on llms.txt, so any benefit today is speculative rather than measured. That doesn't make it worthless, since writing a clear, structured summary of your key content is good hygiene regardless, and if the convention gains traction you're already set up. But don't expect it to move rankings or citations on its own, and don't let it distract from the work that demonstrably matters, like clear content, structured data, and pages that answer real questions well.
If you want to add one, keep it simple. List your most valuable pages with short descriptions, group them logically, and keep it current as your content changes. Think of it as a courtesy to any tool that eventually chooses to use it, filed alongside robots.txt as part of a tidy, machine-readable front door, not as a lever you expect to pull results from this year.
FAQ
What's the difference between GPTBot and OAI-SearchBot? They're both OpenAI's, but they do different jobs. GPTBot fetches content to help train future models, so blocking it keeps your pages out of that training data. OAI-SearchBot fetches content live to build the results ChatGPT shows when it searches the web, so it's the one that can cite and link back to you. If you want to stay out of training but remain citable in ChatGPT's answers, you'd block GPTBot and allow OAI-SearchBot.
Will blocking Google-Extended remove me from Google Search? No. Google-Extended only governs whether your content is used for Google's generative AI training and grounding. It has no effect on Googlebot, which is the crawler that indexes you for the regular search results. You can disallow Google-Extended and keep every bit of your organic ranking and indexing, because the two are handled by completely separate systems.
Do AI crawlers actually obey robots.txt? The reputable ones generally do, and the major named bots like GPTBot, ClaudeBot, and PerplexityBot have documented that they respect robots.txt directives. But robots.txt is voluntary, so a scraper that ignores the rules will simply crawl anyway. For bots you truly need to stop, robots.txt is the first layer, and server-side or firewall blocking is the real enforcement.
Should a small blog bother blocking AI training bots? It depends on how you feel about your content feeding models. Blocking costs nothing and doesn't touch your search rankings, so if you'd rather not contribute your work to training data, it's a reasonable choice. If you'd rather maximize any chance of being surfaced by AI tools, leaning open makes sense. There's no ranking penalty either way, so it comes down to your own preference on how your content gets used.
Does adding llms.txt help me rank or get cited? There's no confirmed evidence that it does yet, because no major AI provider has stated that its crawlers use the file. It's a proposed convention with uncertain adoption. Adding one is harmless and arguably good hygiene, but treat it as an optional experiment, not a tactic with a proven payoff. Your citations and rankings will be decided by your content quality and structured data far more than by this file.
Can I allow some AI crawlers and block others? Yes, and that's the recommended approach. Because each bot uses a distinct user-agent string, you can write a separate rule for each one, blocking the pure training crawlers while allowing the live-answer bots that cite and link. This selective setup lets you keep your content out of raw training sets while staying eligible for the AI answers that actually send you traffic.
Where to Go Next
Start by reading your own logs so you know which AI crawlers are really hitting you and how hard, then decide bot by bot rather than reaching for one blanket rule. Block the pure training crawlers if you'd rather not feed them, opt out of Google-Extended and Applebot-Extended if you want out of AI training, and keep the live-answer bots like OAI-SearchBot and PerplexityBot open so you stay citable.
Remember the core reassurance throughout. None of this touches your Google Search rankings, because AI training tokens and Googlebot are separate systems. Add an llms.txt if you like tidy machine-readable front doors, but spend your real effort on the fundamentals in the robots.txt guide and on being genuinely citable through answer engine optimization. Astro SEO Blog covers crawling, indexing, and AI visibility across the technical SEO category.
Related Articles

How to Fix Cumulative Layout Shift (CLS) in 2026
You go to tap a button and the page jumps, so you tap an ad instead. That's Cumulative Layout Shift, and Google measures it as a ranking signal. Here's exactly what causes it and how to drive your CLS score down to where it should be.

How to Optimize for Bing and Microsoft Copilot in 2026
Almost every SEO guide obsesses over Google and forgets Bing exists. That's a mistake in 2026, because Bing quietly powers Microsoft Copilot, ChatGPT search, and a slice of traffic you're leaving on the table. Here's how to win it.

How to Earn Sitelinks in Google Search
Sitelinks are the extra links Google adds under your listing, and you can't add them manually. Here's how site structure, internal linking, and clear navigation make Google generate them, and why you can only influence, not control, them.