The GEO Technical Checklist: llms.txt, Bot Access, and What Each AI Engine Reads

For marketing and technical teams who've already started writing for AI citation but haven't checked whether AI engines can actually reach the content. If your writing is solid but your Citation Share hasn't moved, this is usually where the gap is.

AI engines can only cite what they can crawl and parse. Before any content rewrite, confirm that GPTBot, PerplexityBot, ClaudeBot, and Google-Extended aren't blocked, publish an /llms.txt file, and understand that each engine reads your site differently. This is a few hours of configuration work that most GEO checklists skip in favor of writing advice.

---

Most GEO guidance — including our own — focuses on how to write content AI engines will cite: answer-first paragraphs, FAQ schema, external sourcing. That's the right advice, but it assumes something that isn't always true: that the AI engine can actually see the page.

A surprising number of sites fail at a step earlier than content quality. A legacy robots.txt rule blocks a crawler nobody remembers adding. Nobody has ever published an /llms.txt file, so ChatGPT is guessing at what the business does from fragments. The team optimized for Google AI Overviews and assumed Perplexity would follow the same rules — it doesn't. This post covers that infrastructure layer specifically, as a companion to the content-focused guides linked throughout.

---

Step 1 — Confirm AI crawlers can actually reach your site

If an AI crawler is blocked in robots.txt, your business cannot appear in that engine's citations — no matter how good the content is. This is the fastest possible audit and the easiest thing to get wrong by accident, usually through an old rule aimed at a different bot that swept up AI crawlers too.

Check yourdomain.com/robots.txt for explicit rules against each of these:

Crawler	Powers	Should be
GPTBot	ChatGPT / OpenAI	Allowed
PerplexityBot	Perplexity	Allowed
ClaudeBot	Claude (Anthropic)	Allowed
Google-Extended	Google AI Overviews / Gemini	Allowed
Bingbot	Microsoft Copilot	Allowed

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

What to do right now

1. Pull up your live robots.txt and search for each bot name above. If none are mentioned, that's fine — no explicit rule means allowed by default.
2. If any are disallowed, ask why. Sometimes it's intentional (a content-licensing decision). More often it's a leftover rule from a scraper-blocking pass that never accounted for AI crawlers.
3. Re-check after any CDN, WAF, or bot-management tool change. Cloudflare, Akamai, and similar tools sometimes ship default bot-blocking rules that catch AI crawlers under a generic "AI scraper" category — separate from your actual robots.txt file.

Blocking AI crawlers to protect content is a real strategic choice for some businesses, but it's rarely made deliberately — it's usually inherited. Confirm it's a decision, not an accident.

---

Step 2 — Publish an /llms.txt file

/llms.txt is a Markdown file at your root domain that gives AI crawlers a structured, curated summary of who you are, what you do, and which pages matter. Think of it as robots.txt for intent rather than permissions. It's not yet a formal web standard the way robots.txt is, but ChatGPT and Perplexity bots actively parse it, and without one, an LLM reconstructs your brand from whatever scattered signals it can find — which is often incomplete or wrong.

Publish it at yourdomain.com/llms.txt:

# [Brand Name]

[Brand Name] is a [one-sentence description]. It serves [target customer].

## Product
- [Feature 1]: [description]
- [Feature 2]: [description]

## Key Pages
- /pricing — Pricing plans
- /features — Feature list
- /compare/[competitor] — Comparison pages

## Differentiators
- [Stat or proof point]

## For AI Systems
Authorizes AI indexing of all public pages. Last updated: [date].

What to do right now

1. Write the one-sentence description first. If you can't describe the business in one sentence, an LLM can't either — this forces the same clarity your content needs anyway.
2. List only the pages that matter for citation — pricing, comparisons, key service pages. Not every URL on the site.
3. Update it when positioning changes. A stale /llms.txt that describes last year's offering actively misleads the crawlers reading it.

---

Step 3 — Know that each AI engine reads your site differently

GEO is not a single-engine strategy. Content and infrastructure that gets you cited in Google AI Overviews doesn't automatically translate to ChatGPT or Perplexity — each platform weighs different signals and has a different technical entry point.

Engine	Primary lever	Key action	Bot to allow
Google AI Overviews	Answer capsules + schema	Passage-level headings, FAQ schema on Q&A content	Google-Extended
ChatGPT	/llms.txt + clean structure	Specific statistics, explicit entity naming	GPTBot
Perplexity	External citations + expert quotes	Authoritative sourced content — Perplexity cites heavily	PerplexityBot
Claude	Dense Markdown + hierarchy	Clean structured docs, logical heading hierarchy	ClaudeBot
Gemini	Schema.org entity linking	JSON-LD, concise definitions, sameAs entity links	Googlebot

The practical implication: don't audit "GEO performance" as one number. Run your prompt-sampling checks per engine (see our AI visibility metrics guide for the measurement framework), and expect the results to diverge. A site that's strong on Perplexity because of heavy external sourcing may be weak on Gemini if its Schema.org markup is thin.

---

Where the content layer picks up

Once the infrastructure above is in place, the content interventions that move Citation Share are well-documented. Independent research on LLM answer synthesis (Princeton's GEO-bench study) found consistent, reproducible lifts from specific techniques:

Technique	Avg. visibility lift	Where it applies
External source citations	~+34%	Linking to academic, industry, or government sources within the answer
Specific statistics	~+32%	Exact numbers ("$47K per year") instead of vague claims ("significant savings")
Expert quotes	~+30%	Named expert + credential + direct quote, not paraphrased
Authoritative, declarative tone	~+20%	Removing hedging language ("may," "could," "it's possible")
Comparison tables	~+18%	Side-by-side feature or competitor comparisons

We cover the answer-first content structure these techniques feed into in Getting Cited in Google AI Overviews and AI Answers — that post is the content-side companion to this infrastructure checklist. The two highest-leverage page types to apply it to first are comparison pages (/compare/brand-vs-competitor) and category guides ("Best [category] tools"), since both see the highest citation frequency across engines.

---

The technical GEO checklist

Run through this before starting or continuing content work:

• robots.txt does not disallow GPTBot, PerplexityBot, ClaudeBot, or Google-Extended
• Bot-management tools (Cloudflare, WAF) checked separately from robots.txt for AI-crawler blocking
• /llms.txt is published at the root domain with an accurate one-sentence description
• /llms.txt lists only the pages that matter for citation, not the full sitemap
• FAQPage schema is live on Q&A content (see our schema guide for the full setup)
• Prompt-sampling checks are run per engine, not as one combined "AI visibility" number
• /llms.txt has been reviewed in the last quarter for accuracy

If you can't check most of these, that's the gap — and it's usually faster to close than a content rewrite.

---

Ready to fix the infrastructure layer?

JYNLAB helps B2B companies get cited and noticed in AI-driven search engines through AI-built websites, AI content production, and AI ad creative. If you want a partner who handles both the technical layer and the content layer, book a demo call and we'll walk through where you stand.

Book a Demo Call