A Shared AI-Training Opt-Out Is Coming to Robots.txt
Matthew Leo · Published October 31, 2026 · Guides
Website owners have spent years maintaining separate robots.txt rules for each company that crawls content for generative AI training. A proposed shared signal could make that job simpler: one policy token, placed in the familiar robots.txt file, that participating providers can interpret across the web.
OpenAI and Google announced support on October 30, 2026, while an Internet-Draft published through the IETF process defines the initial syntax. The important caveat is in the name. This is a draft, not an approved Internet standard, and provider compliance is voluntary.
The site-wide rule
A publisher that wants to express a preference against using any path for generative AI training would add this group to the site’s robots.txt file:
User-Agent: __usage_ai-training
Disallow: /The double underscore is part of the token. The draft also supports the normal robots.txt path model. A site could disallow training generally while explicitly allowing a press kit:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/The draft says the longest matching path wins, following the established robots.txt approach. Existing parsers that do not recognize the token should ignore the group as if it referred to an unknown crawler.
What the signal controls
The token expresses a preference about using retrieved content to modify the learned parameters of a generative AI model. It does not automatically block network access, remove pages from search, prevent quotation, or control every form of machine learning.
The draft explicitly separates a content-use preference from crawler traffic management. If a publisher also wants to block a particular crawler, the site still needs a rule for that crawler’s own user-agent. Existing provider-specific controls therefore remain useful during the transition.
OpenAI says it plans to implement the common signal in the coming months and will continue to honour its existing crawler controls. Google says it plans support in 2027, while retaining Google-Extended. Google also warns that a decision to restrict some uses can affect whether content is eligible for certain Gemini grounding experiences. The implementation details matter as much as the shared line.
A safer rollout checklist
1. Decide the policy before changing the file
Identify which content the organization owns or has authority to license. A blanket rule may be inappropriate for third-party material, public-domain archives, sponsored content or pages covered by separate agreements. Record the business owner and legal rationale for each exception.
2. Keep provider-specific rules for now
Do not delete existing GPTBot, Google-Extended or other provider rules merely because the common token exists. Providers are adopting it on different schedules, and some may never support it. The shared preference is an additional layer, not an instant replacement.
3. Check the file actually served to the public
Fetch https://yourdomain.example/robots.txt from outside the corporate network. Confirm that it returns a successful response, contains plain text, is not blocked by authentication and is not serving an old cached version. Test the apex domain and any important subdomains separately because each host has its own file.
4. Review path precedence
Use a small set of representative URLs: the home page, an article, an image, a paywalled section and any allowed exception. Confirm that the most specific matching rule reflects the intended policy. A misplaced slash can make an exception broader than expected.
5. Monitor provider documentation
Record the date of the change and the version of the draft used. Recheck the IETF document and each provider’s implementation notes before treating the signal as operational. Internet-Drafts can change or expire, and a provider may document proprietary precedence rules.
What this does not settle
A robots.txt preference is not a licence, a contract, a technical access control or a legal judgment. It does not answer whether a particular use is permitted under Canadian copyright law, privacy law or an existing agreement. It also cannot prove that a provider complied.
That limitation is why the proposal is still useful but modest. A common, machine-readable signal reduces maintenance and makes a publisher’s position clearer. It does not create enforcement on its own.
The Verge independently reported that several major AI companies intend to adopt the approach. Publishers should treat that momentum as a reason to prepare, not as evidence that the rule is already universally honoured.
The practical choice for Canadian publishers
Organizations that want to reserve their content from generative AI training can add the shared group after reviewing ownership and exceptions, while keeping current crawler-specific controls. Those that permit training can also express that choice for selected paths. Either way, document the decision, monitor the draft and avoid promising that robots.txt alone provides legal or technical enforcement.
Tags: AI governance, Digital Policy, privacy, web publishing