Blocking AI Crawlers on a Hotel or Tour Operator Website: What GPTBot, OAI-SearchBot, Google-Extended and Your CDN Actually Control

Direct answer: there is no single “block AI” switch, and treating it as one is how travel businesses accidentally disappear from AI answers. Each major AI company now publishes **separate controls for three different jobs**: crawling your pages to train models, indexing them so an AI search product can cite you, and fetching a page live because a traveller asked about it. OpenAI says these settings are independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot so your content is not used for training (OpenAI). Google says Google-Extended does not affect inclusion or ranking in Google Search (Google). So the decision for a hotel group, tour operator, DMC or villa company is really three decisions — plus two checks almost nobody makes: what your CDN or firewall is doing regardless of robots.txt, and whether the facts you want quoted are in the HTML a crawler actually receives.

Kaize is a UK AI product studio for travel and hospitality. This article is based on the crawler documentation published by OpenAI, Google, Anthropic, Apple, Perplexity and Cloudflare as read on 8 October 2026, plus two named pieces of third-party research. It is not a case study: we describe no client results, and we make no claim that any robots.txt setting will win you AI citations or bookings. Crawler names and vendor defaults change; re-check the linked pages before you edit anything.

Why this has landed on your desk now

For years robots.txt was a file a web agency set once and forgot. Two things changed. First, infrastructure providers started shipping AI blocking as a default or a one-click option. On 1 July 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay content creators (Cloudflare), and its documentation now describes AI bot policies split into Search, Agent and Training categories, each set to Block, Block on pages with ads, or Allow — with updated defaults for new domains from 15 September 2026 that block Training and Agent bots on pages displaying ads while leaving Search allowed (Cloudflare docs). Vercel offers a one-click “Block AI Bots” firewall rule (Vercel). Your site may have changed behaviour without anyone in your business deciding it should.

Second, AI products became places where trips are researched. If you are thinking about a ChatGPT app or MCP server for bookings (our ChatGPT app article), the more basic question comes first: can ChatGPT’s, Claude’s, Perplexity’s, Apple’s and Google’s systems read your public pages at all?

Who crawls what: the controls each vendor documents

These are the tokens that matter for a travel website, grouped by vendor and by job. Quotes and behaviours are from each company’s own documentation.

OpenAI (ChatGPT)

• OAI-SearchBot — search. Used to surface websites in ChatGPT’s search features. OpenAI says sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”, and that changes can take about 24 hours to take effect (OpenAI).
• GPTBot — training. Disallowing it indicates your content should not be used to train OpenAI’s generative AI foundation models.
• ChatGPT-User — live fetches when a user asks ChatGPT or a custom GPT something. OpenAI says it is not used for automatic crawling, is not used to decide whether content appears in Search, and “because these actions are initiated by a user, robots.txt rules may not apply.”
• OAI-AdsBot — only visits landing pages submitted as ChatGPT ads, and OpenAI says its data is not used to train foundation models.

Google (Search, AI Overviews, AI Mode, Gemini)

• Googlebot controls Google Search, and Google says AI Overviews and AI Mode are part of Search: “robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search” (Google). There is no separate robots.txt switch for AI Overviews. To limit what is shown, Google points to nosnippet, data-nosnippet, max-snippet or noindex — which affect your ordinary search snippets too.
• Google-Extended is a control token, not a separate crawler. It governs whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI; Google states it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal” (Google).
• Google also says you do not need new machine-readable files, “AI text files”, or special schema.org markup to appear in AI Overviews or AI Mode — a page must simply be indexed and eligible to show with a snippet (Google).

Anthropic (Claude)

• ClaudeBot — collects content that may contribute to model training; restricting it signals your future material should be excluded from training datasets (Anthropic).
• Claude-SearchBot — indexes content to improve search results; Anthropic says disabling it “may reduce your site's visibility and accuracy in user search results.”
• Claude-User — fetches pages when a Claude user asks a question; disabling it prevents retrieval in response to user queries.
• Anthropic says its bots honour robots.txt, that you must set rules on every subdomain you want covered, and that blocking its IP addresses instead “may not work correctly or persistently guarantee an opt-out” because it stops the bot reading your robots.txt.

Apple (Siri, Spotlight, Safari, Apple Intelligence)

• Applebot powers search in Spotlight, Siri and Safari, and its data may also be used to train Apple’s foundation models (Apple).
• Applebot-Extended does not crawl; disallowing it opts your content out of training Apple’s foundation models without removing you from Apple search results. To keep content out of AI-generated “broad world knowledge” answers in Siri and Search, Apple points to the nosnippet meta tag.
• A gotcha for older files: if your robots.txt does not mention Applebot but does mention Googlebot, Apple says Applebot follows the Googlebot instructions.

Perplexity

• PerplexityBot — surfaces and links websites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models (Perplexity).
• Perplexity-User — fetches pages when a user asks a question; because the fetch is user-requested, Perplexity says it “generally ignores robots.txt rules.”

What hotels are actually doing

Independent research by Nicolas Sitter parsed robots.txt files from 105,002 hotel websites across seven countries in March 2026 (Sitter study). Only 3.3% blocked any AI crawler; in the UK sample of 10,547 hotels the figure was 2.0%. GPTBot was the most blocked token at 2.9%, while search bots such as OAI-SearchBot and PerplexityBot were blocked by about 1.0%. The study found 1,190 hotels blocking ChatGPT-User — the token OpenAI says is not used for search inclusion and may not follow robots.txt — and that most blocking was a chain or CMS decision rather than an individual hotel’s: one French cooperative’s shared platform accounted for roughly 27% of all blockers.

Two cautions about that data. It measures robots.txt only, so it cannot see blocks applied at a CDN or firewall. And it is one researcher’s dataset, not an official statistic. Its useful lesson is qualitative: where hotels do restrict AI access, it is usually inherited from a platform, and sometimes aimed at the wrong token.

The three decisions, made deliberately

1. Training: a business choice, not a visibility choice

Whether your copy, guides and photography should be used to train foundation models is a legitimate commercial and rights question. The vendors’ own documentation says opting out of training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) is separate from appearing in their search products (OpenAI; Google; Apple). Our view: most hotel and tour marketing copy exists to be repeated, so many operators will reasonably allow training; a business that has invested in proprietary destination guides or itinerary content may reasonably opt out. Either way, write the choice down and own it.

2. AI search indexing: allow it if you want to be found

If you want ChatGPT, Claude, Perplexity, Apple and Google AI features to be able to cite you, the search-side tokens — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and Googlebot — need access to your public pages. Blocking OAI-SearchBot is, in OpenAI’s words, how you stop being shown in ChatGPT search answers. For a travel business that sells on discovery, that is rarely what anyone intended.

3. User-triggered fetches: mostly not yours to stop

When a traveller pastes your URL into ChatGPT or Perplexity, OpenAI and Perplexity say robots.txt may not apply to that fetch. Claude-User does follow robots.txt. Blocking these agents mainly degrades the answer a prospective guest gets about you. Real risk from automated agents — scraping rates, abusive traffic, or agents acting on your booking engine — is a security and rate-limiting job for your edge, not a robots.txt line (and if you run your own AI agents, see prompt injection controls for the opposite direction of the same problem).

You can also be selective by path. A common, defensible pattern is to keep editorial, room, tour and policy pages open while disallowing endless search-result URLs, account areas and checkout steps for crawlers generally. That protects nothing secret — robots.txt is public — but it stops bots wasting effort on pages that should never be cited.

The dial nobody checks: your CDN and firewall

robots.txt is a request, not an enforcement mechanism. Cloudflare’s own documentation says compliance is voluntary and some operators disregard it (Cloudflare docs). The opposite also happens: an edge rule can block a crawler that your robots.txt explicitly allows. Google’s own AI-features guidance lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” as a basic (Google). Things to check:

• Managed robots.txt. Cloudflare’s managed setting prepends Disallow rules for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and meta-externalagent, plus a content signal of search=yes, ai-train=no, in front of your own file (Cloudflare docs). If your live robots.txt contains lines nobody on your team wrote, this is a likely source.
• AI bot policies by category. Cloudflare classifies verified bots as Search, Agent or Training and lets you block each everywhere, only on pages with ads, or not at all; it also notes that mixed-purpose crawlers used for both search and training are blocked by its training-block configurations (Cloudflare docs). Check which category your important crawlers fall into before assuming “Search is allowed” covers them.
• Allow-list properly if you need to. Perplexity documents WAF allow rules that combine user-agent matching with its published IP ranges (Perplexity); OpenAI publishes IP lists for each of its agents (OpenAI); Google warns that user-agent strings can be spoofed and documents how to verify its crawlers (Google).
• Every hostname. robots.txt is per host. Anthropic explicitly says to set rules on every subdomain (Anthropic). Many hotels and operators run booking engines, brochure sites or blogs on separate subdomains or vendor domains, each with its own robots.txt and its own edge settings.

Cloudflare’s Content Signals (search, ai-input, ai-train) are worth understanding but, as Cloudflare itself says, they express preferences rather than acting as technical countermeasures (Cloudflare).

The second dial nobody checks: what is in your first HTML

Access is pointless if the crawler cannot read what matters once it arrives. Vercel and MERJ’s December 2024 analysis found that none of the major AI crawlers they measured executed JavaScript — including OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot — while Gemini (via Googlebot’s infrastructure) and Applebot did render pages (Vercel). Google itself says server-side or pre-rendering “is still a great idea” because “not all bots can run JavaScript” (Google). The study is nearly two years old and crawler behaviour can change, but the practical test is simple and current: fetch your own page without a browser and look.

Travel sites are unusually exposed here. Room types, cancellation and deposit terms, accessibility information, tour itineraries, inclusions and FAQs often live inside JavaScript booking widgets, iframes or tabs loaded after the page renders. If an assistant cannot see your cancellation policy, it will describe someone else’s — or an OTA’s version of yours. Keeping those facts consistent across your systems is the same discipline as inventory sync and one system of record for bookings and CRM; publishing accessibility details as real text also serves guests using assistive technology (our accessibility article).

A 30-minute audit for a hotel group or tour operator

• Open yourdomain/robots.txt on every hostname you own — main site, booking engine, blog, any brand microsites — and list every AI-related token mentioned.
• For each token, write down which of the three jobs it does (training, search, user fetch) using the vendor pages linked above.
• Flag any Disallow: / under OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Googlebot or User-agent: * that nobody can explain.
• Ask whoever runs your CDN, host or firewall for the current AI bot settings, including any managed robots.txt and category-based blocking, and when they last changed.
• Check server or CDN logs for requests from OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot, and for blocked responses (403s, challenges) served to them; verify genuine bots against the vendors’ published IP ranges.
• Fetch your top room, tour and policy pages without JavaScript (for example with curl) and confirm rate and deposit rules, cancellation terms, inclusions and contact details are present as text.
• Record the decision: what you allow, what you block, why, who owns it, and when you will review it — then check it again after any CDN, CMS or booking-engine change.
• Measure honestly: Google says AI Overviews and AI Mode traffic is included in the Search Console Performance report under Web, so watch that rather than inventing an “AI visibility score”.

An illustrative policy for an operator that wants to be found

This is one defensible shape, not a template to paste. It allows search and user fetches, opts out of model training, and keeps transactional paths out of every crawler’s way. Adapt the paths to your site and test before deploying:

• User-agent: GPTBot / ClaudeBot / Google-Extended / Applebot-Extended — Disallow: / (only if you have decided to opt out of training).
• User-agent: OAI-SearchBot / Claude-SearchBot / PerplexityBot / Applebot / Googlebot — no blanket disallow; same path rules as everyone else.
• User-agent: * — Disallow your checkout, account and internal search-result paths; Allow everything else; list your sitemap.
• At the edge: confirm Search-category crawlers are allowed, decide deliberately on Agent traffic, and keep rate limiting for abusive behaviour regardless of user agent.

Frequently asked questions

If we block GPTBot, do we disappear from ChatGPT?

Not according to OpenAI. GPTBot governs training; ChatGPT search inclusion is governed by OAI-SearchBot, and OpenAI describes the settings as independent (OpenAI).

Does blocking Google-Extended remove us from AI Overviews or AI Mode?

No. Google says AI Overviews and AI Mode are controlled through Googlebot like the rest of Search, and that Google-Extended does not affect Search inclusion or ranking (Google; Google). The only way to limit how your content appears in those features is snippet controls or noindex, which also affect ordinary results.

Do we need an llms.txt file?

Not for Google: it says you do not need new machine-readable files or AI text files to appear in its AI features (Google). Other vendors’ crawler documentation linked here does not ask for one either. Spend the time on robots.txt, edge settings and server-rendered facts first.

Is robots.txt legally binding?

It is a published preference that reputable crawlers say they honour; Cloudflare notes compliance is voluntary (Cloudflare docs). Rights reservations and licensing are legal questions — take advice for your jurisdiction. This article is not legal advice.

Will allowing AI search crawlers get us recommended?

Access is necessary, not sufficient. Being cited depends on whether your pages answer travellers’ questions clearly, consistently and verifiably. No crawler setting guarantees a mention.

What this is not

• Not a case study — no Kaize client, metric or result is described.
• Not a claim that any robots.txt or CDN setting will produce AI citations, traffic or bookings.
• Not an endorsement of Cloudflare, Vercel or any AI vendor.
• Not legal advice on copyright, licensing or data protection.

Kaize point of view

Most travel businesses do not have an AI crawler strategy; they have whatever their CMS, agency or CDN shipped. Our position is simple: **decide by purpose, enforce at the edge, and publish the facts you want quoted in plain HTML.** Training is a rights decision you can make either way. AI search access is a distribution decision, and for an operator that lives on discovery the default should be open. User-triggered fetches are mostly beyond robots.txt, so manage abuse with rate limits rather than blanket blocks. And none of it matters if your cancellation terms are hidden in a widget no crawler can run. This is the plumbing underneath every other AI distribution choice — from NDC, GDS and MCP layers to building your own ChatGPT app.

If you run hotels or a tour operator or DMC and want an outside check of what AI crawlers can and cannot see on your sites — robots.txt, edge rules, booking-engine subdomains and first-HTML content — start with an AI opportunity review or discuss a project with Kaize.