Skip to content
GEOWebsites

Do AI crawlers run JavaScript? What it means for your website

Updated 7 min readBy the Brevity Technologies team

On this page

Googlebot renders JavaScript, but public tests suggest many AI crawlers read only the HTML your server sends, so content that appears only after JavaScript runs may never reach them. The safe approach is server-rendered or static HTML for anything you want found. It takes about a minute to check your own site.

Key takeaways

  • Googlebot renders JavaScript. Public tests suggest many AI crawlers read only the first HTML response.
  • Test any page in a minute with curl, or by turning JavaScript off.
  • Search and training crawlers are separate, so decide on each one individually.
  • Google ignores llms.txt, so server-rendered content and clear, sourced facts matter far more.

What the evidence says

The most cited study is The rise of the AI crawler, published by Vercel and MERJ in December 2024. It analyzed real crawler traffic and reported that major AI crawlers fetched JavaScript files without executing them, with Google's crawling as the main exception. Crawlers change, and these companies don't publish a full list of what they render, so treat this as a good working assumption and test for yourself.

Google is the clear case. Its AI features draw on the Search index, and it says a page must be indexed and eligible to be shown with a snippet to appear in them.

Test your own site in a minute

Pick a sentence that matters, such as your main service description or a price. Then fetch the page without a browser:

curl -s https://yoursite.com/your-page | grep -i "your sentence"

If nothing comes back, that text isn't in the initial HTML. You can check the same thing in a browser by turning off JavaScript or reading View Source. On Chrome's Inspect panel you'd see the rendered page, which is why this problem so often goes unnoticed.

Watch for these common cases:

  • A single-page app whose first response is an empty div
  • Stats that count up from zero, where the HTML contains "0+" and the real figure arrives later
  • FAQ answers or tabs fetched only when someone clicks
  • Testimonials, pricing and case studies loaded from an API in the browser

With Next.js, Nuxt or Astro, you can render these on the server or at build time. Interactive parts can still be client-side, as long as the text is in the first response. Everything on this site is pre-rendered that way, and FAQ answers sit in the HTML even when they're collapsed.

Know which bot does what

Blocking the wrong crawler can take you out of AI search, and allowing the wrong one can give away training data you meant to keep. Each company separates these jobs.

CrawlerPurposeNote
GooglebotGoogle Search, including AI Overviews and AI ModeBlock it and you leave Search
Google-ExtendedA control for Gemini and Vertex AI useDoesn't affect Search inclusion
OAI-SearchBotChatGPT searchAllow if you want to be cited there
GPTBotOpenAI model trainingSeparate decision from search
ChatGPT-UserFetches for a user's requestOpenAI says robots.txt may not apply
PerplexityBotPerplexity searchNot used for model training, per Perplexity
Perplexity-UserFetches for a user's questionGenerally ignores robots.txt
Claude-SearchBotClaude search resultsAllow if you want to be cited there
ClaudeBotAnthropic model trainingSeparate decision from search

Sources for the OpenAI and Perplexity rows are their own documentation pages: OpenAI's crawlers and Perplexity's bots. Also check your CDN or firewall, because a bot-protection setting can block these crawlers even when robots.txt allows them.

What not to spend time on

Google is direct about this. Its documentation says you don't need to create new machine-readable files, AI text files or markup for its AI features, and that llms.txt files "will neither harm nor help" because Google Search ignores them. We add an llms.txt anyway because it takes minutes and other tools may read it, but it isn't a ranking lever.

What does seem to help

The original GEO paper by Aggarwal and colleagues, presented at KDD 2024, tested content changes on a benchmark of 10,000 queries. The best methods beat the baseline by up to 41% on one visibility measure. Adding quotations, statistics and cited sources all performed well, while keyword stuffing did worse than doing nothing. It was a lab benchmark, and results varied by query and by a page's rank, with citing sources helping lower-ranked content far more than top-ranked content.

Bing's guidance for AI answers points the same way, recommending descriptive headings, Q&A sections, tables and structured data, and adding that there's "no secret sauce." Bing's Webmaster Tools also now has an AI Performance report on citations, so you can measure this instead of guessing.

A short checklist

  1. Run the curl test on your homepage, your top service page and your best article.
  2. Move key text, prices and FAQ answers into the initial HTML.
  3. Show real numbers in the markup, not a counter that starts at zero.
  4. Allow the search crawlers you want, and decide separately on training crawlers.
  5. Open each page with a direct answer, with named sources for any statistics.
  6. Check server logs to see which bots actually fetch your pages, and whether they get a 200.

We build these habits into our website development work.

Frequently asked questions

Does ChatGPT run JavaScript when it crawls a website?

A December 2024 study by Vercel and MERJ reported that OpenAI's crawler fetched JavaScript files without executing them. Crawlers change and vendors don't publish full details, so treat this as a working assumption and put anything you want found in the initial HTML.

Do I need an llms.txt file?

Not for Google, which says these files will neither harm nor help because Google Search ignores them. It takes minutes to add one for other tools, but it isn't a ranking lever.

Which robots.txt rule keeps a site citable in ChatGPT?

Allow OAI-SearchBot, which is OpenAI's search crawler. GPTBot is its training crawler and a separate decision, so you can block GPTBot and still be citable in ChatGPT search.

Sources

  1. Vercel and MERJ: The rise of the AI crawler
  2. Google Search Central: AI features and your website
  3. Google Search Central: AI optimization guide
  4. OpenAI: overview of crawlers
  5. Perplexity: crawlers
  6. Aggarwal et al.: GEO, Generative Engine Optimization (KDD 2024)
  7. Microsoft: optimizing content for AI search answers
  8. Bing: AI Performance in Webmaster Tools

Checked on November 2, 2026. Rates and rules change, so confirm them at the source before relying on them.

Keep reading.

All insights
AutomationOdoo

How to automate support email with AI on Odoo and Microsoft 365

Read post
WhatsAppHospitality

WhatsApp chatbots for hotels: upsell flows guests don't mind

Read post

Tell us what's slowing your team down.

One call is usually enough to tell whether software is the answer, and what to build first if it is.