Block AI Training or Get Found by AI? The New Opt-Out

You run a small shop, a studio, a practice or a portfolio site. On 30 October 2026 four AI companies said they will honor, or plan to support, one line in a file on your website that tells them not to train their models on your pages. The short version is “one line opts you out of four companies”. Read the companies’ own posts and it is less than that, and at one company it is more of a trade.

This guide is for the person who has to decide. It covers what the line is, who honors it today (almost nobody yet), what each company’s own pages say you give up if you block, and a way to decide that doesn’t need you to read an Internet-Draft. I did read the draft, and the four announcements, and I quote them where it matters.

One honest note first. I didn’t change a robots.txt on a live site for this guide, and I haven’t tested any vendor’s crawler. What I did run is a small script on three sample robots.txt files, described below, to show one trap. Everything else comes from pages I opened on 31 October 2026. The one exception is a search of X, which I describe where it comes up.

TL;DR. The new line is User-Agent: __usage_ai-training followed by Disallow: /. It says “don’t use my pages to train generative AI models”. It is voluntary and it is only a draft. Anthropic says it has started implementing it; OpenAI says “the coming months”; Google and Microsoft say 2027. It does not stop crawling, it does not remove anything already collected, and a User-agent: * rule does not count as saying it. Blocking training is a separate choice from blocking AI search and answers at OpenAI, Anthropic and, from 2027, Microsoft. At Google the training token is also tied to Gemini’s answers. Decide what you want first. Then keep your existing per-bot rules while the rollout happens.

What was announced on 30 October

Four companies said so in public on the same day. Three of them did it in a post on their own site, and the fourth in a message to the standards group.

Google’s Search Central blog, written by John Mueller, says: “Today, we’re announcing our plans to support a shared AI training control in robots.txt.” The idea is a shared “policy token” in your robots.txt file that sets broad preferences across participating AI providers. The post ends: “We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress.”

OpenAI’s post says it “will honor a shared signal in robots.txt” for how a site’s content is used to train generative AI foundation models, and calls it a voluntary initiative. It says it aims to implement the signal in the coming months, with details in a later update, and that the new signal will not replace the robots.txt controls OpenAI already has. I read the German-language version of the page, so the wording of that sentence is my translation.

Microsoft’s Bing blog, written by Krishna Madhavan, says it is “building on a shared industry training preference” and adding something of its own: a separate AI Grounding preference. Its dates are the most specific of the four. From 1 February 2027 Microsoft says it will honor the new preferences as it crawls.

Anthropic’s statement is a message, not a blog post. Kevin Kelley, the author of the draft, wrote to the IETF working group list on 30 October: “I’m also excited to share that Anthropic has already started implementing support for this draft.” He adds: “It’ll take us some time to add support across our services”. The message names no date and doesn’t mention Anthropic’s crawler, ClaudeBot, so I can’t tell you that ClaudeBot reads the new line today.

So the honest status on 31 October is this.

The posts say “will honor”, “plans to support” and “aims to support”. None of them says “honors it now”, and the difference matters if you were planning to write one line tonight and be done.

What the line actually says

The draft is called “Machine-Readable Training Preference Signal”. It is an Internet-Draft, version 00, dated 30 October 2026 and expiring on 3 May 2027. The draft itself says it is “inappropriate” to cite Internet-Drafts as reference material other than as “work in progress”. Its intended status is “Standards Track”, which says where it hopes to end up, not where it is.

Its whole example is three lines:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That tells a participating AI company: don’t use my pages to train, except the press kit, where training is fine. In the draft’s words it “expresses a preference against AI training for all paths except those under /press-kit/”.

The draft defines AI training as “using an asset to modify the learned parameters of a generative AI model”, and it leaves out models used only for non-generative work. It introduces “no new directive names and no new grammar”. It is the robots.txt you already know, with a name in the User-Agent line that isn’t a real crawler.

Six rules in the draft matter for a site owner.

• The name must match exactly. The draft says to match it “exactly, apart from case”. Two underscores come first, then usage, an underscore, ai, a hyphen and training. A near miss is a different name and says nothing.
• A wildcard does not count. Section 4.2 says wildcard groups “MUST NOT be interpreted as matching the policy token”. Your User-agent: * rules are not an opt-out of training. More on this in the example below.
• No group with the token means no preference. The draft’s words: “no preference has been expressed”. It is not a statement that training is allowed.
• It works by path. Disallow: / is the whole site. Disallow: /portfolio/ is one section. Allow: carves an exception.
• It is not a crawling rule. Section 4.4: “This policy token is not a directive for crawling behavior.” Your existing rules for named crawlers keep working as before, and the draft calls the two things “independent and orthogonal”.
• Specific beats general, and deals beat both. Specific expressions “SHOULD take precedence over general ones”, and the draft says “Private agreements MAY override preferences expressed via this mechanism.”

Microsoft adds one more thing of its own. It lets you set the training preference on a single page too, with a meta tag, and says that a page-level setting overrides the robots.txt line.

A sample you can run

The wildcard rule is where I’d expect most people to trip, so I tested it. On 31 October 2026 I ran Python 3.14.6’s standard urllib.robotparser on three sample files, asking whether each of three names could fetch the path /shop/. These are my own samples, not anyone’s real file.

# File A
User-agent: *
Disallow: /private/

# File B
User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

# File C
User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read the middle row carefully. In file B the standard Python parser says the token name “cannot fetch”, because it treats the wildcard group as applying to every name. The draft says a consumer must not do that. So under the draft, file B has expressed no training preference at all, even though it blocks every crawler that obeys it. File C is the only one where the token is doing its job, and it leaves GPTBot untouched: no rule names it.

That is the point of the whole exercise: a site that has blocked “everything” with a wildcard hasn’t opted out of training with the new line, and a site that has opted out with the new line has not, by that line alone, blocked any crawler.

Training, grounding and search are three different jobs

When people say “block AI”, they usually mix up three things.

• Training: the company uses your pages to modify a generative model’s learned parameters, which is the draft’s definition.
• Search or retrieval: the company’s bot fetches or indexes your pages so a search feature can show them or link to them.
• Grounding: the system looks up pages at the moment of an answer and uses them to write it. Microsoft’s post puts it this way: “Grounding connects generated responses to web content when an answer is created”.

The shared line speaks only to the first. Microsoft’s post says it directly: training and grounding “are independent preferences”, and your training setting “does not change your grounding setting, and vice versa”.

Here is what each company’s own pages say about the controls you have, and what blocking costs.

OpenAI’s own documentation is the clearest. A site can allow OAI-SearchBot to appear in ChatGPT search and disallow GPTBot for training. The page also says: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”. And ChatGPT-User, the bot that visits when a person asks ChatGPT to open a page, is “not used to determine whether content may appear in Search”, and OpenAI says robots.txt rules may not apply to it because the visit is user-initiated.

Anthropic’s help page has the same three-way split. ClaudeBot collects content that could contribute to training. Claude-User fetches a page when a person asks Claude something; disabling it “can reduce your site’s visibility in user-directed web search”. Claude-SearchBot improves search quality; disabling it stops indexing for search and may reduce visibility in search responses. That page is dated 7 April 2026 and doesn’t mention the new line.

Google is the one with a catch

Google-Extended is not a crawler. Google’s page calls it “a standalone product token”, used to manage whether content Google crawls may be used for training future Gemini models, and also for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. The same page says it “does not impact a site’s inclusion in Google Search” and is not a ranking signal.

Now put that next to Mueller’s post. Once Google supports the shared line, a site that opts out of training with it, and does not name Google-Extended, Google plans to treat it “as if the publisher had disallowed Google-Extended”. And “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”

Put plainly, at Google the switch that says “don’t train on me” and the switch that says “show me in Gemini’s answers” are the same switch. Google’s own blog post doesn’t say that sentence. It follows from the two statements above. If you want Gemini to be able to cite and link to you, you leave Google-Extended allowed, and you accept training at Google. If you disallow it, you opt out of both.

This is the part the “one opt-out for everything” headlines skip. I found no independent measurement of how many clicks Gemini Apps send to sites that allow Google-Extended compared with ones that don’t, so I can’t tell you how big the cost is. I can tell you that Google ties them.

Microsoft separates them, with a warning

Microsoft’s new design splits the two cleanly. Training is the shared line, in robots.txt or on a page. Grounding is “a separate page-level preference from Microsoft” with three values: Allow, DisallowExtended and Disallow. There is no site-wide grounding setting in robots.txt.

Both default to Allow when you publish nothing. Microsoft’s example is the combination most small sites would want to look at first: “set AI Training to Disallow and AI Grounding to Allow to keep content eligible to inform generated responses without allowing it to train generative models.”

And then the warning, in Microsoft’s words: “If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences.” Followed by: “Neither choice guarantees inclusion, traffic, or payment.”

DisallowExtended is the middle option. It keeps your content available for answers in Microsoft’s own products, which the post lists as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes the grounding services Microsoft provides to third parties. Neither setting changes your eligibility for ordinary Bing results.

If you use NoArchive or NoCache today, there’s a deadline. Microsoft says to publish the new settings before 1 March 2027, because after that date those two tags “will no longer be read as a preference for training or grounding”. For grounding you also record a temporary choice in Bing Webmaster Tools, which applies per page until the page is recrawled or 30 September 2027, whichever comes first. Search Engine Roundtable’s summary of the post gives 15 December 2026 as the day that temporary setting opens.

How to decide in four questions

You don’t need to understand tokens to decide. You need to know what you sell, and who finds you.

Question 1: do customers find you through AI answers? Don’t guess. Ask ChatGPT, Gemini and Copilot the questions your customers ask, and see whether you or a competitor is named. If you appear, the search and answer bots are working for you, and you should not block them. If you never appear, there is little visibility to lose by blocking, and the bigger task is getting named at all.

Question 2: is the content itself what you sell? A photographer’s images, an illustrator’s portfolio, a course creator’s lesson text, a writer’s essays. If another company’s model learning to imitate them undercuts your income, a training opt-out is worth its small cost. If your pages are menus, hours and a contact form, there is little to protect and a lot to gain from being read.

Question 3: which company matters most to your customers? If your customers use Gemini, the Google trade above applies. If they use ChatGPT, you can block training and still be in its search. If they use Copilot or Bing, you can set training and grounding independently from February 2027.

Question 4: write it, and keep what you have. Add the new line. Keep your existing rules for named bots while the companies roll the signal out. Put a date in your calendar to look again. OpenAI says it will publish implementation details later, and Google says it will provide updates.

Nothing in that table is a rule. The vendors’ pages tell you how the controls work, and none of them tells you what’s right for your business. The line I would hold hardest is the last row: if a page must not be read, don’t rely on any preference. Squarespace’s help page makes the same point: to keep content from crawlers entirely, it says, make it private.

A worked example: Marlow Ceramics

To make this concrete I invented a business. Marlow Ceramics sells handmade mugs and plates from a small website and teaches weekend classes. Its owner cares about two things: that people searching for “handmade mugs near me” can find the shop, and that her glaze photographs aren’t used to teach image models to imitate her. Everything below is a made-up example of the process, not a record of a real site.

Step 1: read what you have. She opens yourdomain.com/robots.txt in a browser. On a site builder it’s generated for her. Hers contains one group, User-agent: *, with a few private paths disallowed. That is file A from the test above. It says nothing about training.

Step 2: answer the four questions. Do customers find her through AI answers? She asks three assistants “best handmade mug shop in Portland”, and her shop is named by one of them. So she keeps the search and answer bots allowed. Is the content the product? Partly: the photos are. Which company matters? Her customers use Google and ChatGPT most, so she reads Google’s trade carefully.

Step 3: choose. At Google she decides that being able to appear in Gemini’s answers is worth more than the opt-out, so she doesn’t name Google-Extended as disallowed. That means Google may use her pages for training, and that is the cost she’s accepting. At OpenAI and Anthropic she can opt out of training without touching search, so she does.

Step 4: write it. She keeps her existing * group and adds these groups:

User-Agent: __usage_ai-training
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Allow: /

The first group is the new shared line. The next two are the per-company training controls that exist today, so the opt-out works before the shared line is implemented. The last group names Google-Extended explicitly, so that once Google supports the shared line, it doesn’t treat her opt-out as a Google-Extended disallow. Google’s post says that reading applies “(and Google-Extended is not explicitly specified)”.

She doesn’t add anything for OAI-SearchBot, Claude-SearchBot or Googlebot. With no rule for them, her * group is what applies, which is what she wants.

Step 5: check it, then wait. She loads her robots.txt again to make sure the lines are there, and spells the token exactly: two underscores, usage, an underscore, ai, a hyphen, training. She puts a date in her calendar for February 2027, when Microsoft says it starts honoring its preferences. She has not changed anything about grounding at Microsoft, so the default, Allow, stands.

What she gets is a set of requests, not a guarantee. What she doesn’t lose is her place in search and in the answers she cares about. And she can see the trade she made at Google, because she wrote it down.

What this means for you

If you run a local business. Your first job is being found, so leave the search and answer bots alone. Ask three assistants a question a customer would ask, and write down who gets named. Then decide about training as a side question. Courses on this are linked at the end.

If you sell on Etsy, Shopify or a marketplace. You may not control robots.txt at all, because the platform does. Check whether your platform has a setting and what it covers. Don’t assume that a setting on your own domain covers your listings on someone else’s.

If you’re a photographer or illustrator. The new line is a statement of preference about training, nothing more. It doesn’t stop a crawler that ignores robots.txt, and it doesn’t pull images out of a model that already learned from them. Write the line, keep the per-bot rules, and treat watermarks and lower-resolution previews as separate decisions.

If you’re a blogger or writer. The choice is partly about values. Opting out of training and staying in search and answers is possible at OpenAI, Anthropic and Microsoft. At Google it is the trade above.

If you manage a company website. Don’t paste the line on a Friday afternoon. Ask who owns robots.txt, what rules already exist for GPTBot, ClaudeBot and Google-Extended, and whether legal wants a say. Microsoft’s NoArchive and NoCache date is a real deadline for you if you use them.

If you use a site builder. The setting in your dashboard may not be the new line yet. Squarespace’s help page, last updated 8 January 2026, predates the announcement. It describes a box labeled “Block known artificial intelligence crawlers”, which adds robots.txt rules for about 28 named crawlers, and which is unchecked by default. I didn’t find anything from Squarespace about the new token on 31 October. Check your builder’s page for a newer date before you assume it’s covered.

Edge cases and troubleshooting

These are the problems I’d expect people to hit, from the draft, the vendors’ pages and the reaction on X.

“I blocked everything with User-agent: * and I’m done.” Under the draft you haven’t expressed a training preference. Your wildcard also blocks obedient crawlers, search included, unless you give them their own groups. The test above shows the difference. Add the token group.

“I wrote the token and nothing changed.” Expected. Anthropic says it has started, OpenAI says the coming months, Google and Microsoft say 2027. Skeptics on X ask whether the promise survives contact with the crawlers that actually run. Keep your per-bot rules in the meantime. Keeping them was one of five themes in Grok’s summary of the X posts, and OpenAI says its existing controls keep applying.

“I blocked Google-Extended and now Gemini doesn’t mention me.” Google ties Google-Extended to grounding in Gemini Apps, and says sites that specifically allow it are eligible for answers and clicks. If being named in Gemini matters, allow it, and say so explicitly.

“I added the shared line and my Google results changed.” Google’s page says Google-Extended does not affect inclusion in Search, and Microsoft says its preferences don’t change Bing result eligibility. If your search traffic fell, look at what else changed. Be careful about edits to a group for Googlebot or Bingbot, which are crawlers, not training controls.

“Microsoft’s grounding setting isn’t in my robots.txt.” Correct. Grounding is page-level only. A site that wants to keep content out of generated answers has to put the setting on each page. A site that only wants to opt out of training can use robots.txt.

“Cloudflare says it handles this.” Cloudflare has a Disallow AI Training option, which Search Engine Journal reported on 15 September 2026. It adds a no-training preference in robots.txt while leaving Googlebot, Applebot and Bingbot able to crawl for search, and maps to Google-Extended at Google. As of that article, before the 30 October announcement, it “won’t send Bing a no-training preference through robots.txt just yet”, and Bing’s training opt-out was the NOARCHIVE tag. I didn’t read Cloudflare’s own page.

“I typed the token and it still doesn’t work in a checker.” A parser that follows the old robots.txt rules will treat * as matching everything, as the Python test did. The token itself is just a name. Check that it starts with two underscores and has no spaces inside it.

“I have no robots.txt.” Microsoft says both of its preferences default to Allow when you publish nothing. I didn’t find the defaults stated on the other three companies’ pages, so for them treat “no file” as no preference expressed.

What this can’t do

• It can’t take back what’s already collected. The draft binds the preference to the robots.txt in effect when a page was retrieved. Nothing in it removes content from datasets or models already built. Squarespace’s page says the same about its own checkbox.
• It can’t make anyone comply. The initiative is described by OpenAI as voluntary. Robots.txt has always been a request. Squarespace says crawlers aren’t required to comply.
• It can’t pay you or license you. Microsoft says these preferences “do not establish licensing or payment arrangements”, and that neither choice guarantees inclusion or traffic. Opting out creates no deal.
• It can’t hide pages. It’s not a directive for crawling, in the draft’s words. A page that has to stay private belongs behind a login.
• It can’t be relied on to stay the same. It is a draft that expires on 3 May 2027 unless revised. Google calls its own plan an aim. The names, rules and dates can change.

FAQ

What does __usage_ai-training mean? It’s a name used in a robots.txt group, not a real crawler. A group that starts with User-Agent: __usage_ai-training states your preference about AI training for the paths that follow.

Does it block ChatGPT from my site? No. It speaks only to training. Whether ChatGPT search can show you depends on OAI-SearchBot, a separate setting. OpenAI’s documentation says each setting is independent.

Will it hurt my Google rankings? Google’s page says Google-Extended doesn’t impact a site’s inclusion in Google Search and isn’t a ranking signal. Google’s 30 October post doesn’t discuss Search. Microsoft says its preferences don’t change eligibility for Bing results.

Do I need to do something today? Only if you want to. Anthropic says it has started implementing the line; the others say later. If you opt out, the per-company controls such as GPTBot and ClaudeBot work now. If you use NoArchive or NoCache with Bing, you have until 1 March 2027.

What is grounding? It means a system looks up your page at the moment of an answer and uses it to write that answer. Microsoft’s post defines it as connecting generated responses to web content when an answer is created.

Is it legally binding? OpenAI calls it voluntary. Google’s post says the proposal is consistent with the EU Code of Practice for general-purpose AI model providers, which recognizes robots.txt as a primary method for publishers to express machine-readable preferences. Microsoft says the preferences set no licensing or payment terms. I’m not a lawyer, so ask one if you have a contract or a rights question.

Can I opt out of one company and not others? Yes. The draft lets you name other crawlers and mixes them with the token. The more specific rule is supposed to win. Google’s post says its own controls keep working alongside the shared one.

Does Squarespace, Wix or WordPress add it for me? I couldn’t confirm that. Squarespace’s help page, updated in January 2026, describes a box for known crawlers. I didn’t verify WordPress or Wix settings for this guide.

Is this the same as turning off training in ChatGPT or Claude? No. This guide is about crawlers reading your website’s pages, which is what the draft covers. The training settings inside a chat app are a different switch, and AI Privacy 101 (linked below) covers them.

The bottom line

The headline is one line of text, and the honest version is a little longer. It’s a good, small idea: say it once, and every participating company can read it. It isn’t live everywhere yet, it isn’t enforced, and it doesn’t reach back. Meanwhile your per-bot rules still do the work.

The real decision hasn’t changed. You’re weighing being read against being used. At OpenAI, Anthropic and, from 2027, Microsoft, you can split those. At Google, today, they’re tied together. Decide which you value, write it down, and look again when the companies ship their versions.

If your next question is how to get AI assistants to name your business in the first place, that’s the other half of this decision. Answer Engine Optimization for Small Business (first 2 lessons free) has you run a five-prompt diagnostic across ChatGPT, Perplexity and Gemini in its second lesson, to see who gets cited. AI Visibility for Local Businesses (first 2 lessons free) covers the local version. For the other side, the chat apps you use yourself, AI Privacy 101 (first 2 lessons free) has a lesson, lesson 3, on turning training off in ChatGPT and Claude and what opting out does not protect you from.

Start lesson 1 free →

Your first step tonight is small: open your own robots.txt, read it once, and write down which of the four questions you couldn’t answer.

Sources

• Kelley, Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00, 30 October 2026
• Kelley, message to the IETF AI Preferences list, 30 October 2026
• Google Search Central Blog, Supporting a shared AI training control for websites, 30 October 2026
• Google, Google’s common crawlers (Google-Extended)
• OpenAI, helping websites express AI training preferences, 30 October 2026
• OpenAI, Overview of OpenAI crawlers
• Microsoft Bing Blog, Your content, your choice: clearer preferences for training and grounding, 30 October 2026
• Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Squarespace Help, Block known artificial intelligence crawlers
• Search Engine Journal, Cloudflare lets sites disallow AI training without blocking Googlebot
• Bing Webmaster Blog, Announcing new options for webmasters to control usage of their content in Bing Chat, 22 September 2023
• Search Engine Roundtable, Google also adopts new AI training control
• Search Engine Roundtable, Microsoft Bing AI controls