static-html-for-ai › Reference

How to Optimize Static HTML for AI Crawlers

By Michael McGrory, Solutions Engineer, Partnerships at Cloudflare Last updated GEOLLMOStatic HTML

Key takeaways

  • Most AI crawlers do not execute JavaScript. Static HTML or server-side rendering is the baseline requirement.
  • The pages most likely to be cited are original research, definitions, comparisons, and how-tos — in that order.
  • GPTBot raw requests grew +305% YoY May 2024 to May 2025; ChatGPT-User grew +2,825%; PerplexityBot grew +157,490% (Cloudflare, 2025).
  • Add llms.txt, a markdown mirror per page, JSON-LD, and Content Signals in robots.txt.
  • Recency matters: AI assistants shift cited publication dates forward by up to 4.78 years when reranking, per a 2025 study.

If you want a static HTML page to be crawled and cited by AI assistants like ChatGPT, Claude, Perplexity, and Google's AI Overviews, you need three things in place: (1) the page content has to live in the initial HTML response, not behind JavaScript; (2) the page has to be structured so that a language model can extract a complete answer to a specific question from a single chunk; and (3) the site has to declare its preferences and signals to crawlers via robots.txt, sitemap.xml, and (increasingly) llms.txt. This page is itself an example of all three.

The three kinds of AI crawlers

AI crawlers are not a single thing. They fall into three categories, and each has a different appetite. Optimizing for one type alone leaves citations on the table.

AI crawler taxonomy — sources: OpenAI, Anthropic, Perplexity, Cloudflare AI Crawl Control reference (2026)
CategoryExamplesHonors robots.txt?What they want
Training GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Google-Extended Yes Breadth, novelty, original data — bulk corpora
Index / search OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot Yes Stable URLs, structured pages, fresh content
User-triggered ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher Largely no Authoritative pages on demand, for citations

If your goal is appearing in AI answers, the index/search and user-triggered bots are what produce visible citations. The training bots build long-term recognition of your brand and topic authority.

Page types most likely to be crawled and cited

Public studies by Ahrefs, Semrush, and Microsoft converge on the same ranking. The list below reflects citation lift, not raw crawl volume.

  1. Original research and proprietary data. Surveys, benchmarks, statistics with a methodology blurb and a sample size. AI assistants need numbers to cite.
  2. Definitional / glossary pages (“What is X”). Direct answer in the first paragraph, then context.
  3. Comparison pages (“X vs Y”, “best X for Y”). Tables and side-by-sides extract verbatim.
  4. How-to and step-by-step guides with numbered steps and code blocks where relevant.
  5. Pricing or cost pages with concrete numbers in plain text (not images).
  6. FAQ and Q&A pages using clear question headings.
  7. Reference and API documentation — this is where llms.txt earns its keep.
  8. Programmatic pages with consistent schemas (e.g. “{thing} statistics 2026”).
  9. Recent news and time-stamped analysis for YMYL or fast-moving topics.
  10. Free tools and calculators — cited heavily and convert well from referrals.

Page-level signals to add

Each item below corresponds to something in this page's HTML. View source if you want a copy-paste reference.

Site-level signals to add

Gotchas worth flagging

JS-rendered SPAs are mostly invisible. If your page needs hydration to render the article body, AI bots see an empty shell. Test with curl -A "GPTBot" https://example.com/page on a sample URL and check that the content is there.
Aggressive bot management blocks legit AI bots. Common mistake on sites that turned on bot fight mode without exceptions. Allowlist verified bots by user-agent and published IP ranges. OpenAI, Anthropic, and Perplexity all publish JSON IP lists.
robots.txt is voluntary. User-triggered fetchers ignore it by design. If you actually need enforcement, use Cloudflare AI Crawl Control or WAF rules.
Stale URLs kill citations. A page cited in ChatGPT today needs to be at the same URL six months from now when a user clicks the source link.

A static HTML checklist for AI crawl

Use this as a final pre-deploy review.

Frequently asked questions

Do AI crawlers execute JavaScript?

Most production AI crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended) do not execute JavaScript. Content rendered client-side is invisible to them. Use server-side rendering, static site generation, or prerendering to put your content in the initial HTML response.

Should I use llms.txt?

Yes, for documentation, API reference, and content-heavy sites. The llms.txt file is a markdown index at the root of your domain that points language models to LLM-friendly versions of your most useful pages. It coexists with robots.txt and sitemap.xml. Adoption is uneven across crawlers, but the cost of adding it is near zero.

Which AI crawlers should I allow in robots.txt?

If your goal is to be cited in AI answers, allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, and ChatGPT-User. Allow GPTBot, ClaudeBot, and Google-Extended only if you also want your content used for model training. Note that ChatGPT-User, Claude-User, and Perplexity-User are user-triggered and largely ignore robots.txt by design.

What page format gets cited most often?

Original research with concrete numbers, definitional or glossary pages, comparison pages, and how-to guides with numbered steps. Pages with quotes and statistics show roughly 30–40% higher visibility in AI answers, according to a 2023 study of 10,000 queries.

Sources