Company announcements October 30, 2026; IETF Internet-Draft dated October 30, 2026
One Line to Opt Out of AI Training: OpenAI, Google, Microsoft and Anthropic Back a Shared robots.txt Token
OpenAI, Google, Microsoft, and Anthropic said on October 30 that they will honor a single shared robots.txt signal that lets a website operator opt content out of AI training across all four crawlers in one entry, instead of maintaining separate rules for each provider crawler. The mechanism is a new user-agent block, User-Agent: __usage_ai-training, followed by Disallow and Allow lines, so a publisher can block training use of the whole site or carve out sections such as a press kit. The signal is defined in an individual Internet-Draft at the IETF titled Machine-Readable Training Preference Signal, authored by Kevin Kelley of Anthropic, dated October 30, 2026 and expiring May 3, 2027. An individual draft is an early proposal, documented and open for review, not an adopted standard. The token governs training use only: it does not prevent crawling altogether and does not remove content from search results, and Google says it will read the signal as applying to its Google-Extended training crawler. It also cannot undo past scraping; content a crawler already collected stays collected, and the signal controls future collection. Timelines differ by company: Anthropic has already begun honoring the token, OpenAI says it will implement the signal in the coming months, Microsoft targets February 1, 2027, and Google targets early 2027. The announcements build on the same voluntary convention robots.txt has always been: a machine-readable preference that participating crawlers agree to respect, now consolidated so one preference reaches four training programs at once.
Executives
If your organization publishes content it does not want in training sets, this turns a per-vendor chore into one maintained line, and the decision belongs in your AI governance record: which sites and sections are opted out, which are deliberately allowed, and who owns the file. Note the limit in the same record: this is a voluntary signal honored by four named companies on their own timelines, not a legal right and not a recall of anything already collected.
CIO / CISO
Add the token decision to the same review that covers your existing robots.txt and crawler rules, because the new block supplements rather than replaces provider-specific controls, and OpenAI states existing rules stay in place. Sequence the work by the published timelines: Anthropic honors it now, Microsoft from February 1, 2027, Google in early 2027, and OpenAI in the coming months, so a file updated today takes effect vendor by vendor. Under the NIST AI RMF Govern function, the durable artifact is the recorded preference and its owner, not the line itself.
Engineers
Implement it exactly as specified: a User-Agent: __usage_ai-training group with Disallow: / for a full opt-out, plus Allow lines for any sections you intend to keep available for training, and validate the file syntax after the change so a formatting error does not silently void the preference. Keep your existing per-crawler groups intact, and re-check the file after any site redesign or content management migration, which is when robots.txt entries most often get overwritten.
Everyone (non-IT)
If you run a website or a blog, you will soon be able to tell four of the largest AI companies, with one small file, not to use your pages to train their models. It will not remove anything they already have, and it does not hide your site from search, it only sets your preference for future training use in a form their crawlers have agreed to read.