Back to Blog

Every AEO thread on LinkedIn is the same. "Add GPTBot to robots.txt." "Block Google-Extended." "Protect your content from AI." The advice moves fast, the source is usually someone else's summary, and the quiet part nobody says out loud is: nobody has actually checked what those blocks do.

So we checked. Two things at once.

First, we audited what Calgary businesses are actually blocking. We fetched /robots.txt for 403 Calgary business homepages and parsed every AI-crawler token against the current documentation from each operator. Second, we opened OpenAI, Anthropic, Perplexity, Google, Meta, DuckDuckGo, Apple, Amazon, Mistral, and You.com's own live docs on the same day and mapped what each block actually does. The two studies together tell a story that has almost nothing to do with the LinkedIn version.

What We Did

Corpus: 403 unique Calgary business homepages, drawn from 27 "best of Calgary" listicles across nine verticals (restaurants, gyms, cafes, contractors, dental, med-spa, law, auto services, mortgage brokers, retail). Full source list and exclusion policy are in the methodology section below.

For each site we fetched /robots.txt, followed up to 5 redirects, and parsed the file per RFC 9309 (contiguous User-agent lines grouped with the following rules). We then evaluated whether each of 22 current AI crawler tokens was blocked, either explicitly or via a wildcard User-agent: *. We also tracked 3 legacy tokens (anthropic-ai, claude-web, FacebookBot) that still appear in the wild but are no longer in the operators' current documentation.

333 of 403 sites (82.6%) served a readable robots.txt. The 70 sites without one allow every crawler by default. The audit script is at tools/bulk-robots-audit.js and the per-site JSON is at data/blog/ai-blocking-myth-calgary-2026/raw/. The full anonymized dataset is at data/blog/ai-blocking-myth-calgary-2026/metrics.csv.

The Headline Number

Of 403 sites, 17 block at least one current AI crawler token. That is it. 96% of Calgary businesses have opted into every AI training corpus, every AI search index, and every live user-retrieval bot on the internet, either deliberately or by not doing anything.

And within that 4.2%: 16 sites block a training-corpus bot (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended). Only 8 sites block a live user-retrieval bot (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, DuckAssistBot). Six sites use User-agent: * / Disallow: / which blocks every crawler on earth, including Googlebot, which almost certainly is not what the site owner intended.

The tokens blocked cluster together tightly. GPTBot, CCBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, and Bytespider all get blocked on roughly the same 15 to 16 sites. This is a fingerprint of a copy-paste block list making the rounds on Twitter and Reddit. Fifteen Calgary businesses ran that list. Everyone else did nothing.

Token-Level Breakdown

Of the 403 sites, this is how many block each specific bot:

Every training bot gets blocked twice as often as its matching live-retrieval bot. Which is the exact opposite of what most business owners intend, because training already happened on their old content years ago. The bot that decides whether a customer's question about them today gets answered with a link to their site is the live-retrieval bot.

The Truth Table: What Those Blocks Actually Do

This is the part almost no one gets right. We opened each operator's current documentation on 2026-07-20 and quoted their own words on what each token does.

OpenAI. Blocking GPTBot does not remove you from ChatGPT. GPTBot is a training crawler, not a retrieval crawler. OpenAI states plainly that the two settings are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot." Live browsing when a user asks ChatGPT is ChatGPT-User. Appearance in ChatGPT's search surface is OAI-SearchBot. If you want to reduce ChatGPT citation you block ChatGPT-User and OAI-SearchBot. Blocking GPTBot alone does nothing to your presence in ChatGPT answers.

Google. Blocking Google-Extended does not remove you from AI Overviews or AI Mode. Google's crawler doc says explicitly: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode draw on the regular Google Search index, which is built by Googlebot. The only lever Google documents for suppressing your snippet in AI Overviews is nosnippet, data-nosnippet, or noindex, and those also suppress your ordinary Search snippets. Restated: there is no robots.txt token that removes you from AI Overviews without also removing you from Google Search itself. Every Calgary site that added Google-Extended: Disallow thinking it opts out of AI Overviews got exactly nothing for it.

Perplexity. Blocking Perplexity-User is best-effort. Blocking PerplexityBot removes you from the Perplexity index. But Perplexity's own docs say Perplexity-User "generally ignores robots.txt rules" because "a user requested the fetch." A robots.txt block does not reliably prevent live citation.

Anthropic. The clean case. Blocking ClaudeBot is training only. Blocking Claude-User "prevents our system from retrieving your content in response to a user query" per Anthropic. Blocking Claude-SearchBot handles search. Anthropic represents that all three obey robots.txt. So Anthropic is the one operator whose live-retrieval opt-out actually works.

Meta. Mostly honors it, with a live-fetch exception. Search inclusion in Meta AI is Meta-WebIndexer. But meta-externalfetcher, the user-request fetcher, "may bypass robots.txt rules" per Meta's own doc.

DuckDuckGo. Disallowing DuckAssistBot removes you from AI-assisted answers. Clean, per DDG's help page.

ByteDance. Bytespider has no operator documentation and is widely reported to ignore robots.txt entirely. You cannot rely on blocking it.

Net rule: a robots.txt block reliably suppresses training and search-index crawlers, but for live user-initiated retrieval only Anthropic and DuckDuckGo cleanly honor it. Perplexity-User and Meta-ExternalFetcher may bypass it. Bytespider ignores it. Google has no live-retrieval AI bot to block at all.

The Platform Story

The next question is: are these blocks even a choice, or does the site's builder ship them by default? We opened each platform's own help centre on 2026-07-20 to check.

No platform blocks AI crawlers by default. Wix, Squarespace, Shopify, WordPress.com, Webflow, and Duda all allow every AI crawler out of the box. Squarespace and WordPress.com are the only two with a purpose-built one-click AI-blocking toggle. Squarespace's toggle, when enabled, disallows 26 named AI bots at once and ships OFF by default. WordPress.com's "Prevent third-party sharing" checkbox is similar and also ships OFF. Everywhere else you edit robots.txt manually.

This means the 17 Calgary sites blocking AI made an intentional choice, or someone at their web-design shop did. This is not a case of platforms silently opting people out. The Calgary market is 96% opt-in to AI, essentially by inaction. Squarespace + WordPress.com combined represent about 48% of the corpus, and even those users had to actively flip the switch.

We also detected one site whose block pattern matched the fingerprint of Squarespace's 26-bot bulk toggle (six or more current tokens blocked plus the legacy anthropic-ai that Squarespace ships in the bundle). So of the 17 blockers in Calgary, one used the Squarespace toggle. The other 16 hand-rolled a robots.txt.

What to Actually Do

Split the question into two separate levers. Do not confuse them.

Lever one: do I want my content used to train future AI models. If no, block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, and Amazonbot. This has zero effect on whether you get cited today. Your existing content is already in training corpuses from years past. This is about future model versions, not current answers.

Lever two: do I want AI engines to cite me when a customer asks about me. For most Calgary businesses the honest answer is yes, because AI citations are how a growing share of new customers find you. If you want the citations, allow ChatGPT-User, Claude-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, and Amzn-SearchBot. Do not accidentally block them with a broad User-agent: * Disallow rule.

And separately, do the AEO work that actually gets you cited. Blocking or unblocking a crawler does not put you in AI answers. Being a clearly structured, well-marked source of the answer does. Add valid LocalBusiness or Organization schema. Ship an llms.txt. Add FAQPage schema on pages with real questions and answers. Give the crawlers something worth citing.

A Sample robots.txt for a Calgary Business That Wants Citations

Copy and paste, edit the sitemap URL, drop at /robots.txt. This is what "opt into being cited, opt out of training" looks like for a business that wants ChatGPT, Claude, Perplexity, and Google to answer with a link to your site. Every disallow below is a training crawler policy. Every allow is a retrieval crawler, the one that fetches your page when someone actually asks a question.

Note what is not here. Any User-agent: * / Disallow: / rule at the top. Any block on Googlebot. Any belief that blocking GPTBot alone opts you out of ChatGPT. Those are the three moves that quietly hurt Calgary businesses today.

Methodology

Sample. 403 unique Calgary business homepages, drawn from 27 "best of Calgary" industry listicles verified 2026-07-20. Verticals covered: restaurants, gyms, cafes, contractors, dental, med-spa, law, auto services, mortgage brokers, retail. Real estate is represented by mortgage brokers because every Calgary realtor listicle links to internal profile pages rather than to independent websites. Full source list, exclusion policy (30+ hosts and aggregators removed), and vertical mapping are in data/blog/ai-blocking-myth-calgary-2026/method.json.

Selection bias. Read the sample as "Calgary businesses polished enough to be listicle-featured and to own a linkable site." The frame over-represents established, well-marketed businesses that already invest in a strong web presence. Any homepage-quality finding here is an upper bound. The true Calgary SMB population, including newer, cash-only, and referral-driven businesses, is likely worse on every measure.

Audit tool. Custom Node script at tools/bulk-robots-audit.js. Each site's /robots.txt was fetched with a 12-second timeout, 5-hop redirect limit, and a 500 KB size cap. The file was parsed per RFC 9309 (contiguous User-agent lines share the following rules; blank lines and other directives end a group). A site was recorded as blocking a token if any group whose User-agent list matched that token (case-insensitive) contained a Disallow: / or Disallow: /* rule with no more-specific Allow: /. Wildcard User-agent: * blocks were counted separately and propagated to every token in the summary.

Token list. 22 current AI crawler tokens across 12 operators (OpenAI, Anthropic, Perplexity, Google, Common Crawl, Apple, Meta, Amazon, Mistral, DuckDuckGo, You.com, ByteDance), plus 3 legacy tokens (anthropic-ai, claude-web, FacebookBot) reported separately. The current-token list is authoritative per each operator's live documentation on 2026-07-20. Where an operator uses a token that only controls a policy signal rather than a distinct crawler (Google-Extended, Applebot-Extended), we treat it as a block target because that is how it is used in practice.

Platform fingerprint. Detected from HTML fingerprints in a parallel audit (server headers, static asset domains, embedded platform paths). Squarespace's one-click AI-blocking toggle was fingerprinted separately as "six or more current AI tokens blocked plus the legacy anthropic-ai token," because the legacy token is included in Squarespace's bundle and almost never appears in a hand-rolled robots.txt.

Limitations. Homepage /robots.txt only. Some multi-site operators use different robots policies on different subdomains. We did not check meta robots tags on individual pages, and we did not evaluate X-Robots-Tag HTTP headers, which can suppress indexing without touching robots.txt. Truth-table findings reflect each operator's stated behavior on 2026-07-20; operator behavior changes and this study should be re-run periodically.

Naming policy. No site is named in this study. Whether a business blocks or does not block AI is a defensible choice either way, and naming laggards on either axis is not fair game.

Replication. The full anonymized dataset is at data/blog/ai-blocking-myth-calgary-2026/metrics.csv. The audit script is at tools/bulk-robots-audit.js. Run it against any URL list and the JSON output format matches this study exactly.

Related Reading

• The State of AEO in Calgary 2026, the companion AI-search readiness study.
• llms.txt explained, what the optional discovery file does and does not do.
• How to get cited by ChatGPT, the operator playbook that starts with allowing the right bots.
• How to rank on Perplexity, since Perplexity is the one live engine that bypasses robots.txt anyway.

Frequently Asked Questions

Does blocking GPTBot remove my site from ChatGPT citations?

No. GPTBot is a training crawler, not a retrieval crawler. OpenAI's own documentation states the two settings are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot." Live browsing when a user asks ChatGPT is a different bot called ChatGPT-User. Appearance in ChatGPT's search surface is OAI-SearchBot. To reduce the chance of ChatGPT citing your site you must block ChatGPT-User (live retrieval) and OAI-SearchBot (search), not GPTBot. The common belief that "block GPTBot = out of ChatGPT" is wrong per OpenAI's own docs.

Does blocking Google-Extended remove my site from AI Overviews or AI Mode?

No. Google-Extended is a control token that manages whether your content is used to train future Gemini models. Google's crawler documentation says explicitly: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode draw on the regular Google Search index (Googlebot). The only lever Google documents for suppressing your snippet in AI Overviews is nosnippet, data-nosnippet, or noindex, and those also suppress your ordinary Search snippets. There is no robots.txt token that removes you from AI Overviews without also removing you from Search.

How many Calgary businesses actually block AI crawlers?

17 out of 403 (4.2%). 16 sites (4%) block at least one training-corpus bot such as GPTBot, ClaudeBot, or CCBot. Only 8 sites (2%) block any live user-retrieval bot such as ChatGPT-User or Claude-User. Six sites use User-agent: * with Disallow: / and block everything. Almost every other Calgary site allows every AI crawler by default. Data collected 2026-07-20.

Do platforms like Wix or Squarespace block AI crawlers for me by default?

No. We verified each platform's own documentation on 2026-07-20. Wix, Squarespace, Shopify, WordPress.com, Webflow, Duda, and every platform whose default behavior is documented all allow AI crawlers out of the box. Squarespace and WordPress.com are the only two with a purpose-built one-click AI-blocking toggle, and both ship with it OFF by default. Everywhere else you must edit robots.txt manually (or, on Shopify, ship a robots.txt.liquid override, which Shopify labels an unsupported customization).

Which AI crawlers actually respect robots.txt for live retrieval?

Per each operator's own documentation on 2026-07-20: Anthropic represents that Claude-User obeys robots.txt. DuckDuckGo says DuckAssistBot obeys it. Perplexity states plainly that Perplexity-User "generally ignores robots.txt rules" because "a user requested the fetch." Meta says meta-externalfetcher "may bypass robots.txt rules." ByteDance has no operator documentation and Bytespider is widely reported to ignore robots.txt entirely. Google has no live-retrieval AI bot at all; AI Overviews use Googlebot's index. So even a correctly configured robots.txt only cleanly stops Claude and DuckAssist from citing you live. The others either bypass it or are unaffected.

What should a Calgary business actually do?

Decide the question you are actually answering. There are two different levers, do not confuse them. Lever one: "do I want my content used to train future AI models." If no, block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, and Amazonbot. Zero effect on whether you are cited today. Lever two: "do I want AI to cite me when a customer asks about me." The honest answer for most Calgary businesses is yes, because AI citations are how new customers are increasingly finding you. If yes, ALLOW ChatGPT-User, Claude-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, and Amzn-SearchBot. Do not block them by accident with a wildcard rule. And separately, actually do the AEO work: add valid LocalBusiness schema, add an llms.txt file, add FAQPage schema on pages with real Q&A. That is what makes you a citable source. Blocking or unblocking a crawler does not put you in AI answers. Being a clearly-marked, structured source of the answer does.