The AI-blocking myth: 403 Calgary sites later, almost no one blocks AI. And the ones who do picked wrong.
We fetched robots.txt for 403 Calgary business homepages on July 20, 2026, and evaluated every block against each operator’s own live documentation. Only 4.2% of Calgary businesses block any AI crawler at all. Only 2% block a live-retrieval bot. Six sites accidentally block everyone with a wildcard rule. And the popular advice everyone follows (“block GPTBot to opt out of ChatGPT”, “block Google-Extended to stay out of AI Overviews”) is wrong per each operator’s own docs. Almost no one is blocking anything, and the few who are picked the wrong bots.
Quote any figure above with a link to this page. Cite it as Meridian15 (2026). The AI-Blocking Myth: 403 Calgary Sites Later, Almost No One Blocks AI (and the Ones Who Do Picked Wrong). https://fifteenthmeridian.com/blog/ai-blocking-myth-calgary-2026. The full result table is metrics.csv and the method is method.json.
Every AEO thread on LinkedIn is the same. “Add GPTBot to robots.txt.” “Block Google-Extended.” “Protect your content from AI.” The advice moves fast, the source is usually someone else’s summary, and the quiet part nobody says out loud is: nobody has actually checked what those blocks do.
So we checked. Two things at once.
First, we audited what Calgary businesses are actually blocking. We fetched /robots.txt for 403 Calgary business homepages and parsed every AI-crawler token against the current documentation from each operator. Second, we opened OpenAI, Anthropic, Perplexity, Google, Meta, DuckDuckGo, Apple, Amazon, Mistral, and You.com’s own live docs on the same day and mapped what each block actually does. The two studies together tell a story that has almost nothing to do with the LinkedIn version.
What we did
Corpus: 403 unique Calgary business homepages, drawn from 27 “best of Calgary” listicles across ten verticals (restaurants, gyms, cafes, contractors, dental, med-spa, law, auto services, mortgage brokers, retail). Full source list and exclusion policy are in the methodology section below.
For each site we fetched /robots.txt, followed up to 5 redirects, and parsed the file per RFC 9309 (contiguous User-agent lines grouped with the following rules). We then evaluated whether each of 22 current AI crawler tokens was blocked, either explicitly or via a wildcard User-agent: *. We also tracked 3 legacy tokens (anthropic-ai, claude-web, FacebookBot) that still appear in the wild but are no longer in the operators’ current documentation.
333 of 403 sites (82.6%) served a readable robots.txt. The 70 sites without one allow every crawler by default. The audit script is at tools/bulk-robots-audit.js. The full anonymized dataset is at /research/blog/ai-blocking-myth-calgary-2026/metrics.csv.
The headline number
4.2%
of Calgary businesses block any AI crawler
Of 403 sites, 17 block at least one current AI crawler token. That is it. 96% of Calgary businesses have opted into every AI training corpus, every AI search index, and every live user-retrieval bot on the internet, either deliberately or by not doing anything.
And within that 4.2%: 16 sites block a training-corpus bot (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended). Only 8 sites block a live user-retrieval bot (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, DuckAssistBot). Six sites use User-agent: * / Disallow: / which blocks every crawler on earth, including Googlebot, which almost certainly is not what the site owner intended.
The tokens blocked cluster together tightly. The six wildcard sites block every token, so they count toward each total. Beyond those, nine sites block the same eight tokens: GPTBot, CCBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, and Bytespider. That looks like one shared block list copied from site to site. Two more sites block a few tokens each. Everyone else did nothing.
Token-level breakdown
Of the 403 sites, this is how many block each specific bot:
Every training bot gets blocked at least twice as often as its matching live-retrieval bot. That runs against what most business owners want if the goal is to be cited today, because the bot that decides whether a customer’s question about them today gets answered with a link to their site is the live-retrieval bot.
The truth table: what those blocks actually do
This is the part almost no one gets right. We opened each operator’s current documentation on 2026-07-20 and quoted their own words on what each token does.
OpenAI. Blocking GPTBot does not remove you from ChatGPT. GPTBot is a training crawler, not a retrieval crawler. OpenAI states plainly that the two settings are independent: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot.” Live browsing when a user asks ChatGPT is ChatGPT-User. Appearance in ChatGPT’s search surface is OAI-SearchBot. If you want to reduce ChatGPT citation you block ChatGPT-User and OAI-SearchBot. Blocking GPTBot alone does nothing to your presence in ChatGPT answers.
Google. Blocking Google-Extended does not remove you from AI Overviews or AI Mode. Google’s crawler doc says explicitly: Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” AI Overviews and AI Mode draw on the regular Google Search index, which is built by Googlebot. The only lever Google documents for suppressing your snippet in AI Overviews is nosnippet, data-nosnippet, or noindex, and those also suppress your ordinary Search snippets. Restated: there is no robots.txt token that removes you from AI Overviews without also removing you from Google Search itself. Every Calgary site that added Google-Extended: Disallow thinking it opts out of AI Overviews got exactly nothing for it.
Perplexity. Blocking Perplexity-User is best-effort. Blocking PerplexityBot removes you from the Perplexity index. But Perplexity’s own docs say Perplexity-User “generally ignores robots.txt rules” because “a user requested the fetch.” A robots.txt block does not reliably prevent live citation.
Anthropic. The clean case. Blocking ClaudeBot is training only. Blocking Claude-User “prevents our system from retrieving your content in response to a user query” per Anthropic. Blocking Claude-SearchBot handles search. Anthropic represents that all three obey robots.txt. So Anthropic is the one operator whose live-retrieval opt-out actually works.
Meta. Mostly honors it, with a live-fetch exception. Search inclusion in Meta AI is Meta-WebIndexer. But meta-externalfetcher, the user-request fetcher, “may bypass robots.txt rules” per Meta’s own doc.
DuckDuckGo. Disallowing DuckAssistBot removes you from AI-assisted answers. Clean, per DDG’s help page.
ByteDance. We found no operator documentation for Bytespider, so there is no stated policy to rely on. You cannot count on blocking it.
Net rule: a robots.txt block reliably suppresses training and search-index crawlers, but for live user-initiated retrieval only Anthropic and DuckDuckGo cleanly honor it. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user started the action. Perplexity-User and Meta-ExternalFetcher may bypass it. Google lists user-triggered fetchers, including Gemini Notebook and Google-Agent, that Google says can ignore robots.txt rules because a user requested the fetch.
The platform story
The next question is: are these blocks even a choice, or does the site’s builder ship them by default? We opened each platform’s own help centre on 2026-07-20 to check.
No platform blocks AI crawlers by default. Wix, Squarespace, Shopify, WordPress.com, Webflow, and Duda all allow every AI crawler out of the box. Squarespace and WordPress.com are the only two with a purpose-built one-click AI-blocking toggle. Squarespace’s toggle, when enabled, adds a list of known AI crawlers to robots.txt at once and ships OFF by default. WordPress.com’s “Prevent third-party sharing” checkbox is similar and also ships OFF. Everywhere else you edit robots.txt manually.
This means every one of the 17 blocks was added by the site owner or their web-design shop, deliberately or, in the six wildcard cases, probably by accident. This is not a case of platforms silently opting people out. The Calgary market is 96% opt-in to AI, essentially by inaction. None of the 19 Squarespace sites in the corpus blocked any AI crawler.
One site matched our fingerprint for a bulk blocking toggle (six or more current tokens blocked plus the legacy anthropic-ai token), but our platform detection did not identify it as a Squarespace site, so we cannot say which tool wrote it. None of the other 16 blockers shows the signature.
What to actually do
Split the question into two separate levers. Do not confuse them.
Lever one: do I want my content used to train future AI models. If no, block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, and Amazonbot. This has zero effect on whether you get cited today. A block applies from the day you add it and does nothing about content collected before then. This is about future crawling, not current answers.
Lever two: do I want AI engines to cite me when a customer asks about me. For most Calgary businesses the honest answer is yes, because a citation puts a link to your site in the answer a customer is reading. If you want the citations, allow ChatGPT-User, Claude-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, and Amzn-SearchBot. Do not accidentally block them with a broad User-agent: * Disallow rule.
And separately, do the AEO work that actually gets you cited. Blocking or unblocking a crawler does not put you in AI answers. Being a clearly structured, well-marked source of the answer does. Add valid LocalBusiness or Organization schema. Add FAQPage schema on pages with real questions and answers. Give the crawlers something worth citing.
A sample robots.txt for a Calgary business that wants citations
Copy and paste, edit the sitemap URL, drop at /robots.txt. This is what “opt into being cited, opt out of training” looks like for a business that wants ChatGPT, Claude, Perplexity, and Google to answer with a link to your site. Every disallow below is a training crawler policy. Every allow is a retrieval crawler, the one that fetches your page when someone actually asks a question.
Note what is not here. Any User-agent: * / Disallow: / rule at the top. Any block on Googlebot. Any belief that blocking GPTBot alone opts you out of ChatGPT. Those are the three moves that quietly hurt Calgary businesses today.
Methodology
Sample. 403 unique Calgary business homepages, drawn from 27 “best of Calgary” industry listicles verified 2026-07-20. Verticals covered: restaurants, gyms, cafes, contractors, dental, med-spa, law, auto services, mortgage brokers, retail. Real estate is represented by mortgage brokers because every Calgary realtor listicle links to internal profile pages rather than to independent websites. Full source list, exclusion policy (30+ hosts and aggregators removed), and vertical mapping are in /research/blog/ai-blocking-myth-calgary-2026/method.json.
Selection bias. Read the sample as “Calgary businesses polished enough to be listicle-featured and to own a linkable site.” The frame over-represents established, well-marketed businesses that already invest in a strong web presence. Any homepage-quality finding here is an upper bound. The true Calgary SMB population, including newer, cash-only, and referral-driven businesses, is likely worse on every measure.
Audit tool. Custom Node script at tools/bulk-robots-audit.js. Each site’s /robots.txt was fetched with a 12-second timeout, 5-hop redirect limit, and a 500 KB size cap. The file was parsed per RFC 9309 (contiguous User-agent lines share the following rules; blank lines and other directives end a group). A site was recorded as blocking a token if any group whose User-agent list matched that token (case-insensitive) contained a Disallow: / or Disallow: /* rule with no more-specific Allow: /. Wildcard User-agent: * blocks were counted separately and propagated to every token in the summary.
Token list. 22 current AI crawler tokens across 12 operators (OpenAI, Anthropic, Perplexity, Google, Common Crawl, Apple, Meta, Amazon, Mistral, DuckDuckGo, You.com, ByteDance), plus 3 legacy tokens (anthropic-ai, claude-web, FacebookBot) reported separately. The current-token list is authoritative per each operator’s live documentation on 2026-07-20. Where an operator uses a token that only controls a policy signal rather than a distinct crawler (Google-Extended, Applebot-Extended), we treat it as a block target because that is how it is used in practice.
Platform fingerprint. Detected from HTML fingerprints in a parallel audit (server headers, static asset domains, embedded platform paths). Squarespace’s one-click AI-blocking toggle was fingerprinted separately as “six or more current AI tokens blocked plus the legacy anthropic-ai token,” because the legacy token is included in Squarespace’s bundle and almost never appears in a hand-rolled robots.txt.
Limitations. Homepage /robots.txt only. Some multi-site operators use different robots policies on different subdomains. We did not check meta robots tags on individual pages, and we did not evaluate X-Robots-Tag HTTP headers, which can suppress indexing without touching robots.txt. Truth-table findings reflect each operator’s stated behavior on 2026-07-20; operator behavior changes and this study should be re-run periodically.
Naming policy. No site is named in this study. Whether a business blocks or does not block AI is a defensible choice either way, and naming laggards on either axis is not fair game.
Replication. The full anonymized dataset is at /research/blog/ai-blocking-myth-calgary-2026/metrics.csv. The audit script is at tools/bulk-robots-audit.js. Run it against any URL list and the JSON output format matches this study exactly.
Related reading
- The State of AEO in Calgary 2026, the companion AI-search readiness study.
- llms.txt explained, what the optional discovery file does and does not do.
- How to get cited by ChatGPT, the operator playbook that starts with allowing the right bots.
- How to rank on Perplexity, since Perplexity’s live fetcher can ignore robots.txt anyway.
Questions, answered.
Does blocking GPTBot remove my site from ChatGPT citations?
No. GPTBot is a training crawler, not a retrieval crawler. OpenAI's own documentation states the two settings are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot." Live browsing when a user asks ChatGPT is a different bot called ChatGPT-User. Appearance in ChatGPT's search surface is OAI-SearchBot. To reduce the chance of ChatGPT citing your site you must block ChatGPT-User (live retrieval) and OAI-SearchBot (search), not GPTBot. The common belief that "block GPTBot = out of ChatGPT" is wrong per OpenAI's own docs.
Does blocking Google-Extended remove my site from AI Overviews or AI Mode?
No. Google-Extended is a control token that manages whether your content is used to train future Gemini models. Google's crawler documentation says explicitly: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode draw on the regular Google Search index (Googlebot). The only lever Google documents for suppressing your snippet in AI Overviews is nosnippet, data-nosnippet, or noindex, and those also suppress your ordinary Search snippets. There is no robots.txt token that removes you from AI Overviews without also removing you from Search.
How many Calgary businesses actually block AI crawlers?
17 out of 403 (4.2%). 16 sites (4%) block at least one training-corpus bot such as GPTBot, ClaudeBot, or CCBot. Only 8 sites (2%) block any live user-retrieval bot such as ChatGPT-User or Claude-User. Six sites use User-agent: * with Disallow: / and block everything. Almost every other Calgary site allows every AI crawler by default. Data collected 2026-07-20.
Do platforms like Wix or Squarespace block AI crawlers for me by default?
No. We verified each platform's own documentation on 2026-07-20. Wix, Squarespace, Shopify, WordPress.com, Webflow, Duda, and every platform whose default behavior is documented all allow AI crawlers out of the box. Squarespace and WordPress.com are the only two with a purpose-built one-click AI-blocking toggle, and both ship with it OFF by default. Everywhere else you must edit robots.txt manually (or, on Shopify, ship a robots.txt.liquid override, which Shopify labels an unsupported customization).
Which AI crawlers actually respect robots.txt for live retrieval?
Per each operator's own documentation on 2026-07-20: Anthropic represents that Claude-User obeys robots.txt. DuckDuckGo says DuckAssistBot obeys it. Perplexity states plainly that Perplexity-User "generally ignores robots.txt rules" because "a user requested the fetch." Meta says meta-externalfetcher "may bypass robots.txt rules." OpenAI says robots.txt rules may not apply to ChatGPT-User. We found no operator documentation for Bytespider. Google documents user-triggered fetchers, including Gemini Notebook and Google-Agent, that Google says can ignore robots.txt, and AI Overviews use Googlebot's index. So even a correctly configured robots.txt only cleanly stops Claude and DuckAssist from fetching your pages live. The others either bypass it or are unaffected.
What should a Calgary business actually do?
Decide the question you are actually answering. There are two different levers, do not confuse them. Lever one: "do I want my content used to train future AI models." If no, block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, and Amazonbot. Zero effect on whether you are cited today. Lever two: "do I want AI to cite me when a customer asks about me." The honest answer for most Calgary businesses is yes, because a citation puts a link to your site in the answer a customer is reading. If yes, ALLOW ChatGPT-User, Claude-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, and Amzn-SearchBot. Do not block them by accident with a wildcard rule. And separately, actually do the AEO work: add valid LocalBusiness schema and add FAQPage schema on pages with real Q&A. That is what makes you a citable source. Blocking or unblocking a crawler does not put you in AI answers. Being a clearly-marked, structured source of the answer does.
Faster site. Then AI search. One sprint.
If your Calgary site is in the Slow or Painful tier, we fix speed and AEO together. Same weekend, same team, both wins.
