Who Took My Clicks? The State of SEO and AI in 2025 - Chapter 3

Pillar 3: Technical Foundations — Crawlability, Data & Trust

“The unseen tech checklist that decides whether AI can even see your tours.”

Why This Pillar Matters

When Google’s AI Overview or ChatGPT compiles an answer, it can only quote what it can crawl, parse, and trust. If an LLM’s crawler can’t read your site—or can’t verify your facts—your brand disappears from the conversation before the first traveller even sees a result. This chapter breaks down the invisible plumbing that keeps small‑to‑mid‑sized travel brands discoverable.

Executive Checklist

Fix these five items this quarter:

  1. Robots open – verify AI bots are allowed and not rate‑limited.

  2. Robots can read your page – ensure View‑Source shows full page text.

  3. Schema added – Itinerary, TouristTrip, Offer, FAQPage on revenue pages.

  4. Bounce audit – remove pop‑up traps, compress images, pass Core Web Vitals.

  5. Publish Markdown alt‑pages for flagship guides.


1 · Let the Bots In

Key idea: AI crawlers announce themselves (via user‑agent strings) just like Googlebot. If you block them in robots.txt, you forfeit citations.

Common AI Crawlers

What They Power

Where to Check

GPTBot (OpenAI)

ChatGPT, SearchGPT

https://yourdomain.com/robots.txt

OAI-SearchBot

OpenAI real‑time search

Same

Google-Extended / GoogleOther

Gemini data collection

Same

ClaudeBot

Anthropic Claude

Same

Travel example: A boutique safari operator blocked GPTBot thinking it saved bandwidth. Suddenly ChatGPT’s answer to “10‑day Zambia safari sample itinerary” no longer cited them—bookings dipped 12 %.

Best‑practice snippet:

User‑agent: GPTBot

Allow: /

User‑agent: Google-Extended

Allow: /

Place this above any generic User‑agent: * rules so it’s not overridden.


2 · Render vs. JavaScript Pitfalls

LLM crawlers read raw HTML. Heavy client‑side JavaScript (React, Vue, Angular) often returns a blank page unless server‑side rendering (SSR) is enabled.

  • SSR delivers fully‑formed HTML from the server—crawlers see every heading, list, and FAQ instantly.

  • CSR (client‑side rendering) requires a browser to build the page; most crawlers skip this expensive step.

Checklist:

  1. Load your page, right‑click → View Page Source. Do you see the full itinerary text? If not, bots don’t either.

  2. If missing, enable SSR (Next.js / Nuxt / hydrogen SSR plug‑in) or add dynamic rendering—serve pre‑rendered HTML to bots.


3 · Structured Data Stacks

Schema markup turns page facts into machine‑readable triples. For travel, four high‑impact types dominate GEO visibility:

Schema Type

Primary Use

Travel Example

Itinerary

Defines multi‑day schedules

12‑day “Andes & Amazon Expedition”

TouristTrip

Describes destination plus organizer

5‑day “Whale Safari – Húsavík”

Offer

Prices, availability dates, currency

“€3 450 per adult, valid 2025‑06‑01 → 2025‑09‑30”

FAQPage

Clear Q‑A pairs for LLM extraction

“Do I need yellow‑fever vaccine?”

Markup snippet (JSON‑LD):

{

 “@context”: “https://schema.org”,

 “@type”: “Itinerary”,

 “name”: “7‑Day Luxury Galápagos Cruise”,

 “description”: “Island‑hopping aboard the M/Y Sea Finch…”,

 “touristType”: “Luxury”,

 “hasPart”: [{

   “@type”: “TouristTrip”,

   “name”: “Day 3 – Española Island”,

   “itinerary”: “…”

 }]

}

Search Console’s Rich Results Test validates your schema and flags errors.


4 · Site Health & NavBoost UX Signals

Google’s internal NavBoost system (revealed in the 2024 API leak) uses bounce‑back data to demote results that disappoint users. High exit rates from your page hurt both classic SEO and AI passage rankings.

  • Keep Core Web Vitals green. Slow mobile pages trigger bounces.

  • Kill intrusive pop‑ups. One study showed a 17 % higher bounce when newsletter modals load within 3 seconds.

  • Deliver what the heading promises. If “Peru Altitude FAQ” clicks through to a generic tour brochure, travellers bounce—and NavBoost notes the deception.


5 · Security & Privacy Considerations

Crawlers respect robots.txt but still store any content they fetch. Avoid exposing:

  • Confidential pricing tiers for B2B agents.

  • Customer PII in reviews.

  • Draft content with outdated facts (could be quoted!).

Tip: Put staging or confidential docs in a subdomain and disallow all bots:

User‑agent: *

Disallow: /

 

6 · Markdown Mirrors for Cleaner Crawls

Early tests show that publishing a Markdown version of key guides—linked in <link rel=”alternate” type=”text/markdown”>—improves passage extraction accuracy. Markdown removes layout clutter, giving models pure headings, lists, and FAQs.

Example:
island‑hopping‑guide.mdx ranks higher in ChatGPT than the same HTML page—because bullet steps and code‑fenced prices are perfectly parseable.

Glossary (Executive‑Friendly)

Term

Meaning for Non‑Tech Leaders

Crawler / Bot

Automated program that reads your website.

robots.txt

File that tells crawlers what they may or may not read.

Server‑Side Rendering (SSR)

Building the full HTML on the server so bots see everything.

Schema / Structured Data

Hidden code that labels your facts clearly.

NavBoost

Google metric that lowers rankings if users bounce quickly.

Markdown

Plain‑text formatting language easy for AI to parse.