Question 1

Do ChatGPT and Perplexity crawlers execute JavaScript when indexing pages?

Accepted Answer

No — GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot do not execute JavaScript when crawling web pages for AI search indexing. They behave like HTTP clients that fetch the raw HTML response from your server and stop there. If your site relies on a client-side JavaScript framework like React, Vue, or Angular to render its content, those crawlers will see only the bare-bones HTML shell — typically an empty div with a script tag. None of the text, headings, product descriptions, FAQs, or structured data that you consider your content will be visible to them. Google's crawlers have executed JavaScript since approximately 2015 using a headless Chromium pipeline, which is why sites built as SPAs often rank fine in traditional search. AI crawlers have made no equivalent investment. As of May 2026, all three major AI search crawlers are confirmed non-JS-executing. This is not expected to change imminently — the compute cost of rendering JavaScript at crawl scale is prohibitive without a major infrastructure commitment.

Question 2

What is the difference between SSR and static site generation for AI crawler visibility?

Accepted Answer

Both server-side rendering (SSR) and static site generation (SSG) solve the AI crawler visibility problem, but they work differently and have different trade-offs. SSR renders each page on the server at request time — when the crawler fetches a URL, it receives fully formed HTML with all content embedded. SSG pre-builds HTML files at deploy time and serves them as static assets from a CDN, which means every crawler fetch returns complete HTML with no server processing required. For AI crawler visibility, SSG is generally superior because it removes all latency from the content delivery path, eliminates any server-side rendering errors, and allows aggressive CDN caching without the cloaking risks of session-based personalization. SSR works well when content must be dynamic per request — authenticated user state, real-time inventory, personalized pricing. For the majority of marketing and content pages that AI assistants actually index and cite, SSG is the right architectural choice. The practical answer for most teams: use SSG for all non-authenticated pages, SSR for authenticated or highly dynamic pages, and ensure both output complete HTML before any JavaScript executes.

Question 3

How do I check if my site is visible to GPTBot and ClaudeBot?

Accepted Answer

There are four practical methods for auditing AI crawler visibility. First, use curl or wget with no JavaScript execution to fetch your pages raw — run curl -s https://yourdomain.com/your-page and inspect the response body. If the body is an empty div or contains only script tags with no meaningful text, AI crawlers see nothing. Second, disable JavaScript in your browser (Chrome DevTools > Settings > Debugger > Disable JavaScript) and load your pages. What you see is approximately what AI crawlers see. Third, use a dedicated crawler simulation tool like Screaming Frog in non-JS mode or the Google Search Console's URL Inspection tool with the 'test live URL' option — though note that GSC uses Google's JS-executing crawler, so it will show more than AI crawlers actually see. Fourth, check your server logs for visits from GPTBot, ClaudeBot, and PerplexityBot user agents, then map those visits against your content pages to identify coverage gaps. The most reliable single-step test remains the curl approach — it directly replicates what non-JS crawlers receive.

Question 4

Does Next.js app router automatically handle AI crawler visibility?

Accepted Answer

Next.js with the App Router handles AI crawler visibility correctly when you use Server Components — which are the default in App Router. Server Components render on the server and return fully formed HTML, meaning AI crawlers receive complete content without needing to execute JavaScript. However, there are two significant failure modes to watch for. First, any component marked with 'use client' that contains meaningful text content will be rendered only in the browser, making that content invisible to AI crawlers. A common mistake is wrapping large content sections in client components for animation or interactivity. Second, dynamic routes that use generateStaticParams or getServerSideProps must be correctly configured to ensure they serve full HTML rather than empty shells during the initial response. The safest approach is to verify each important page type using a raw curl fetch to confirm that all headings, body text, FAQs, and structured data appear in the raw HTML response. Next.js App Router is AI-crawler-safe by default, but that safety is easy to break with improper 'use client' usage.

Question 5

What is the fastest way to fix an AI crawler rendering gap without a full platform migration?

Accepted Answer

If a full migration to SSR or SSG is not feasible in the near term, three interim fixes can reduce the visibility gap without a rewrite. First, implement prerendering middleware that detects known bot user agents — GPTBot, ClaudeBot, PerplexityBot, Googlebot — and serves a pre-rendered static HTML snapshot to those agents while continuing to serve the JavaScript SPA to human visitors. Tools like Prerender.io and Rendertron provide this as a managed service. Be aware that serving fundamentally different content to bots and users can trigger cloaking penalties in traditional search if not implemented carefully. Second, use a service worker or build-time static export to generate HTML snapshots of your highest-priority content pages — specifically product pages, feature pages, comparison pages, FAQ pages, and any page you want cited in AI answers. Third, evaluate which pages drive the most AI citation value and implement targeted SSR for only those pages using framework-specific partial rendering options, such as Next.js route segment configuration. The goal is to get the 20% of pages responsible for 80% of your potential AI citation surface delivering full HTML within one quarter.