SSR vs. CSR for AI crawlers on React, Next.js and Astro

What is actually in the HTTP response?

Open View Source and search for a sentence from the article. Then fetch the same route with curl. If the sentence exists only in the browser DOM after JavaScript runs, a simple HTTP client cannot read it from that response.

A loading spinner, navigation labels and a serialized data object are not equivalent to a readable article. The scanner extracts visible document text and can compare it with an optional browser render. Without a real render, it cannot measure every hydration or client-navigation behavior.

curl -sS 'https://example.com/guide/' -o page.html
# Search the saved response for a sentence from the main answer.
rg -n 'A distinctive sentence from your guide' page.html

• Serve content before client JavaScript
• Interpret a measured rendering gap

Can ChatGPT, Claude and Perplexity run JavaScript?

Do not assume one rendering capability across every search crawl and user-requested fetch. Those are different services and identities, and operators do not promise identical behavior for every request. Google documents JavaScript rendering, but even that does not remove the need for crawlable resources and usable page content.

You can avoid this dependency for most documentation and marketing pages. Send the useful text and links as HTML. Let JavaScript improve the interface after the document is readable. If a private dashboard needs a login, keep it private; there is no reason to expose its data to satisfy a crawler check.

• Google JavaScript SEO basics
• Separate search crawlers from user fetchers

Keep React and Next.js content in the server output

A React page can render on the server or entirely in the browser. In the Next.js App Router, a Client Component can still be prerendered into HTML. The use client directive alone is not evidence of a content gap. Client-only effects and explicitly disabled server rendering deserve closer inspection.

The example keeps a documentation answer in the route component. A separate interactive component can handle a checker, copy button or filter. Adapt data fetching and caching to your installed Next.js version; do not paste configuration from an unrelated router version.

// app/guide/page.tsx
export default function GuidePage() {
  return (
    <main>
      <h1>Which crawler rule applies?</h1>
      <p>The most specific matching robots.txt group controls this path.</p>
      <h2>How can I verify it?</h2>
      <p>Fetch the deployed file and test the exact URL path.</p>
    </main>
  );
}

• Next.js server HTML and client-only rendering

Use Astro islands for the interactive part

Astro renders framework components to HTML by default. A client directive adds hydration for an interaction; client:only skips server rendering of that component. Keep the main explanation outside a client-only island when it should be discoverable in the initial document.

A useful split is a static article with a hydrated calculator below it. Explain the inputs and the calculation in HTML so the page remains understandable even when the interactive code fails to load.

---
import Calculator from "../components/Calculator.jsx";
---
<main>
  <h1>Calculate the required crawl budget</h1>
  <p>Divide the number of public URLs by the planned refresh interval.</p>
  <Calculator client:visible />
</main>

• Astro component rendering and hydration

Verify the published route, not just the source code

Build and deploy the page, fetch its public URL and repeat the sentence check. Check several real content routes rather than just /. A client-side router can serve a polished homepage while deep routes return an empty shell or a fallback document.

Compare signed-out behavior, redirects and the actual canonical URL. Then rescan. Keep a raw-HTML check in the deployment workflow for a few representative routes if future framework changes could move content back into client code.

• Use the scan API in a deployment check