{"id":13484,"date":"2026-11-02T09:32:00","date_gmt":"2026-11-02T07:32:00","guid":{"rendered":"https:\/\/voctos.net\/ai-training-control-robots-txt-google-bing-arab-publishers\/"},"modified":"2026-11-02T09:32:00","modified_gmt":"2026-11-02T07:32:00","slug":"ai-training-control-robots-txt-google-bing-arab-publishers","status":"publish","type":"post","link":"https:\/\/voctos.net\/ar\/ai-training-control-robots-txt-google-bing-arab-publishers\/","title":{"rendered":"\u0636\u0648\u0627\u0628\u0637 \u062a\u062f\u0631\u064a\u0628 \u0627\u0644\u0630\u0643\u0627\u0621 \u0627\u0644\u0627\u0635\u0637\u0646\u0627\u0639\u064a \u0627\u0644\u062c\u062f\u064a\u062f\u0629 \u0645\u0646 Google \u0648Bing: \u0645\u0627 \u0627\u0644\u0630\u064a \u064a\u062c\u0628 \u0623\u0646 \u064a\u0636\u0628\u0637\u0647 \u0627\u0644\u0646\u0627\u0634\u0631\u0648\u0646 \u0627\u0644\u0639\u0631\u0628 \u0642\u0628\u0644 1 \u0645\u0627\u0631\u0633 2027"},"content":{"rendered":"

Google and Microsoft both announced a shared AI training control on 30 October 2026. Microsoft starts honoring it on 1 February 2027 and wants publishers who use NoArchive or NoCache to act by 1 March 2027. Google plans support in 2027 without a date. Of 25 Arab robots.txt files we read on 2 November 2026, none uses it yet.<\/p>\n

What did Google and Bing announce on 30 October 2026?<\/h2>\n

Both companies back a single robots.txt signal, based on the IETF AI Preferences working group, that lets a site opt out of AI training across providers in one step. Google calls it a “policy token” and plans support in 2027. Microsoft published exact syntax and dates the same day.<\/p>\n

Google’s Search Central post<\/a> (John Mueller, 30 October 2026) says that when a site uses the token to opt out of training and does not name Google-Extended, Google reads that as a Google-Extended disallow. Existing company-specific controls keep working next to the shared one. The post does not print the token itself.<\/p>\n

Microsoft’s Bing announcement<\/a> does. The training line is a user-agent called __usage_ai-training<\/em> followed by a Disallow rule, for a whole site or a single path. Bing also adds page-level versions of the same preference through a meta tag or an X-Robots-Tag header. Our source for the industry reaction<\/a> says OpenAI and Anthropic are on board too, but neither company’s own post was published when we checked, so treat that as reported, not confirmed.<\/p>\n

What is the difference between AI training and AI grounding?<\/h2>\n

Training uses your pages to change the parameters of a model. Grounding uses your pages as input while an answer is being written, which is what produces a citation and a click. Bing now lets you set the two independently. Blocking one does not block the other.<\/p>\n

That split matters because the commercial value for an Arab brand sits in grounding. A clinic, a retailer or a publisher earns visibility when ChatGPT, Gemini or Copilot cites its page for an Arabic question, not when its text sits in a training set. Google makes the same link: sites that explicitly allow Google-Extended stay eligible to appear in Gemini Apps responses and receive clicks through grounding.<\/p>\n\n\n\n\n\n\n\n\n
Preference<\/th>\nWhat it controls<\/th>\nWhere you set it<\/th>\nDefault<\/th>\n<\/tr>\n<\/thead>\n
AI training (shared)<\/td>\nUse of content to train generative models<\/td>\nrobots.txt for a site or path; Bing also accepts it per page<\/td>\nAllow<\/td>\n<\/tr>\n
AI grounding (Bing only)<\/td>\nUse of content to inform generated answers<\/td>\nPage level only: meta tag or X-Robots-Tag<\/td>\nAllow<\/td>\n<\/tr>\n
Limited grounding (DisallowExtended)<\/td>\nMicrosoft’s own products only, not third-party grounding services<\/td>\nPage level only<\/td>\nNot set<\/td>\n<\/tr>\n
Google-Extended<\/td>\nGoogle’s AI training control since 2023, also tied to Gemini Apps grounding eligibility<\/td>\nrobots.txt<\/td>\nAllow<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Bing is explicit about the cost. Choosing Disallow or DisallowExtended for grounding reduces your visibility in AI experiences, and none of the choices guarantees inclusion, traffic or payment. Google-Extended does not affect ranking in Google Search or AI Overviews, a point we covered in our 50-brand robots.txt check<\/a>.<\/p>\n

Which dates matter for Arab publishers?<\/h2>\n

Four Bing dates matter, and only one is a deadline. Temporary grounding controls open on 15 December 2026, Microsoft starts honoring the new preferences on 1 February 2027, the deadline for NoArchive and NoCache users is 1 March 2027, and temporary preferences expire on 30 September 2027.<\/p>\n\n\n\n\n\n\n\n\n
Date<\/th>\nWhat happens<\/th>\nWho must act<\/th>\n<\/tr>\n<\/thead>\n
15 December 2026<\/td>\nTemporary grounding preferences open in Bing Webmaster Tools<\/td>\nOnly sites that use NoArchive or NoCache on that date<\/td>\n<\/tr>\n
1 February 2027<\/td>\nMicrosoft honors the new preferences as it crawls<\/td>\nEveryone who wants a non-default setting<\/td>\n<\/tr>\n
1 March 2027<\/td>\nNoArchive and NoCache stop counting as training or grounding preferences<\/td>\nSites that use them and want to keep the restriction<\/td>\n<\/tr>\n
30 September 2027<\/td>\nTemporary grounding preferences expire<\/td>\nSites relying on the Webmaster Tools setting<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

If you do not record a temporary grounding preference by 1 March, pages that carry NoArchive or NoCache default to Allow for grounding until Microsoft recrawls them. Google gives no date beyond 2027. Nothing here needs action this week, but any site that uses noarchive today needs a decision before Q1 2027.<\/p>\n

Want a second pair of eyes on which of your pages AI answers can still cite? Ask us to review your AI search eligibility<\/a> before you change a single line.<\/p>\n

What do 25 Arab robots.txt files say today?<\/h2>\n

Most of them say nothing about AI. On 2 November 2026 we fetched the robots.txt of 36 Arab and regional sites in news, retail, telecom, health and government. Twenty-five returned a readable file. Fourteen name no AI crawler, seven block at least one, four allow named AI bots, and none uses the new token.<\/p>\n\n\n\n\n\n\n\n\n\n
Policy in robots.txt (25 readable files)<\/th>\nSites<\/th>\nExamples<\/th>\n<\/tr>\n<\/thead>\n
No AI crawler named, so the generic rules apply<\/td>\n14<\/td>\nNews portals, a telecom, two government sites, a food-delivery app, a law firm<\/td>\n<\/tr>\n
Blocks at least one AI bot site-wide<\/td>\n7<\/td>\nAl Jazeera (GPTBot, ClaudeBot, PerplexityBot), Khaleej Times (Google-Extended, ClaudeBot), Sabq, Al Moosa Hospital, three Amazon storefronts for Saudi Arabia, the UAE and Egypt<\/td>\n<\/tr>\n
Names AI bots with an explicit Allow or path-level rules<\/td>\n4<\/td>\nstc, Vodafone Egypt, Youm7, Jarir<\/td>\n<\/tr>\n
Uses a Content-Signal line (ai-train=no)<\/td>\n2<\/td>\nSabq, Al Moosa Hospital<\/td>\n<\/tr>\n
Uses the new shared training token<\/td>\n0<\/td>\nNot yet shipped by Google<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

We also read the homepage source of the 17 sites that returned a full page. One, Khaleej Times, carries a noarchive robots meta tag. Under Bing’s plan that tag stops counting as a training or grounding preference after 1 March 2027, so it is the clearest example in our sample of a restriction that needs re-publishing. The other 16 homepages show no noarchive, nocache or AI preference tags.<\/p>\n

The limits are real. We fetched from a cloud server, not from Riyadh, Dubai or Cairo, so a CDN can serve a different file to a local visitor. Eleven sites returned no readable file or blocked our request, and we read homepages only. The sample is a spot check of large brands, not a market census, and it overlaps with the 2 October study<\/a>, which covered 50 brands and found the same pattern: silence is the norm.<\/p>\n

What should you publish, and where?<\/h2>\n

Decide training and grounding separately, then publish only what differs from the default. A site that wants to stay citable and keep its text out of training needs one robots.txt rule today and a per-page tag later. A site happy with both needs nothing at all.<\/p>\n\n\n\n\n\n\n\n\n\n
Your goal<\/th>\nWhat to publish<\/th>\n<\/tr>\n<\/thead>\n
Stay citable and allow training<\/td>\nNothing. Both default to Allow. Keep Googlebot, Bingbot, OAI-SearchBot and PerplexityBot unblocked.<\/td>\n<\/tr>\n
Block training, stay citable<\/td>\nIn robots.txt: User-Agent: __usage_ai-training, then Disallow: \/ (Bing syntax). Keep Google-Extended allowed if you want Gemini grounding.<\/td>\n<\/tr>\n
Block training on one section<\/td>\nThe same user-agent with Disallow: \/section-name\/<\/td>\n<\/tr>\n
Block grounding on a sensitive page<\/td>\nMeta name Bingbot, content ai-grounding:disallow, or the X-Robots-Tag header. Expect lower AI visibility for that page.<\/td>\n<\/tr>\n
Currently use NoArchive or NoCache<\/td>\nReplace with the new tags before 1 March 2027 and record a temporary grounding preference in Bing Webmaster Tools from 15 December 2026.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Two cautions from Bing’s own guide. These settings govern use of content, not crawl access, and an Allow does not override a robots.txt rule that blocks Bingbot. A published setting also does nothing until Microsoft recrawls the page; IndexNow can prompt a recrawl but does not guarantee a date. If a CDN or firewall blocks bots that robots.txt allows, fix that first, as explained in our bot traffic guide<\/a>.<\/p>\n

Our read<\/h2>\n

For most Arab brands the right move is to do nothing to grounding and decide on training by business model. A publisher selling Arabic journalism has a reason to opt out of training. A clinic, retailer or lender whose goal is being recommended has none, and blocking grounding there removes the visibility the content was written to earn.<\/p>\n

We would not rush to add the shared token. Google has not published it and gives no date, so a line written today targets Bing’s syntax only. Add it when Google’s documentation prints the exact token. Treat the Content-Signal line, which two of our 25 sites already use, as a bridge, not a replacement.<\/p>\n

The one task with a hard deadline is the audit. Search your templates for noarchive and nocache. Our sample suggests few Arab sites have the tag, but a site that has it loses the restriction on 1 March 2027 unless it acts. This also touches GEO work for Arabic content<\/a>: Gemini’s share of AI referrals in the Gulf is moving, as our StatCounter October analysis<\/a> shows with its caveats, so a Google-Extended block now carries a cost it did not carry a year ago.<\/p>\n

For proof of what citation eligibility is worth, VOCTOS’s Delta Medical Labs case study<\/a> reports 5,570 pages cited by AI engines as of July 2026. Those pages are only citable because the crawlers behind those engines are allowed in.<\/p>\n

FAQ<\/h2>\n

Do I need to change robots.txt before Google ships the new token?<\/h3>\n

No. Google says it aims to support the shared control in 2027 and existing controls such as Google-Extended keep working alongside it. If you already block Google-Extended, that block stays in force. A new line today only reaches Bing, which publishes its syntax and honors it from 1 February 2027.<\/p>\n

Will blocking AI training remove my site from Google or Bing results?<\/h3>\n

No. Bing states that these preferences do not change eligibility for traditional search results, and Google says Google-Extended does not affect Search ranking. Blocking grounding is different: Bing warns it significantly reduces visibility in AI-generated experiences, so apply it only to pages you do not want cited.<\/p>\n

What happens to my NoArchive tag after 1 March 2027?<\/h3>\n

Microsoft stops reading NoArchive and NoCache as training or grounding preferences after that date. Pages with those tags default to Allow for grounding until recrawled unless you record a temporary preference in Bing Webmaster Tools, which opens on 15 December 2026. Publish the new tags on the pages that matter before March.<\/p>\n

Does the new control make llms.txt or Content-Signal obsolete?<\/h3>\n

Not today. The shared control covers training only, and llms.txt is a different kind of file, as we explained in our llms.txt check<\/a>. Content-Signal is a separate Cloudflare-style line that some sites already publish. Keep it until a provider says it reads the new token.<\/p>\n

How do I check what my own site publishes today?<\/h3>\n

Open yourdomain.com\/robots.txt and search for GPTBot, Google-Extended, ClaudeBot and PerplexityBot. Then view the source of your homepage and article template for noarchive and nocache. Finally, ask your host whether a firewall or CDN blocks AI crawlers, because robots.txt cannot override those rules.<\/p>\n

If your site carries noarchive today, schedule the change before 15 December and test one template first. If it carries nothing, leave grounding alone and check that no firewall is blocking the bots you want. Send us your robots.txt and tell us what you want AI answers to cite<\/a>, and we will point out what it allows today.<\/p>\n