Skip to content
AskVault Docs
Try Free →

How to scrape a JavaScript-rendered website for LLM training

Written by , Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 8 min read

Why HTTP-only scrapers fail in 2026

Around 65% of marketing sites built since 2022 ship as a JavaScript-rendered single-page app. The HTML the server returns is essentially empty:

<!DOCTYPE html>
<html>
  <head><title>Acme Corp</title></head>
  <body>
    <div id="root"></div>
    <script src="/static/main.abc123.js"></script>
  </body>
</html>

The actual content (hero copy, feature list, pricing, testimonials, FAQ) is built at runtime by the JavaScript bundle. A scraper that runs httpx.get(url) and returns the response body gets the empty shell. Then your RAG system embeds the empty shell. Then your AI agent has no idea what your business does.

This is the single most common reason a "smart chatbot" trained on a website returns useless answers. The scraper didn't render JavaScript, so the knowledge base is empty.

The two failure modes you actually have to handle

There are two distinct cases where naive scraping breaks. They need different fixes.

Case 1: JavaScript-rendered content. The site is a single-page app built with React, Vue, Angular, Svelte, or similar. The server returns a shell, the client builds the page. Detected by inspecting the response body: if it is suspiciously small and contains an empty app container (<div id="root"> or similar) with little surrounding text, you need a headless browser.

Case 2: Anti-bot protection. The site is protected by Cloudflare, Akamai, PerimeterX, or similar. The server returns a challenge page ("Just a moment...", "Checking your browser before accessing...") and only serves real content after a JavaScript proof-of-work or a TLS fingerprint check. Detected by status code (403 or 503) or body content (the word "challenge" or "Just a moment").

Most production scrapers conflate these. They escalate to a headless browser in both cases. That works but it's wasteful: a large share of pages don't need a browser at all, and a browser-based fetch is far slower and more expensive than a plain HTTP fetch.

The tiered approach

The right architecture is a tiered fallback: try the cheap fetch first, escalate only when it fails. The shape of it:

  1. Plain HTTP fetch with a normal browser user-agent. Sub-second per page. Works on a large share of modern marketing sites.
  2. Headless browser (for example Playwright or Puppeteer with Chromium). A few seconds per page. Works on JavaScript-rendered SPAs.
  3. Anti-bot bypass service (a premium scraping API). Slower still, but works on Cloudflare-protected and high-friction sites.

Each step escalates only if the previous one returns a known-failure signal. A good crawler also remembers which hosts needed a browser, so repeat requests to that host skip the cheap fetch and start at the browser step. That per-host learning is what turns a slow crawl into a fast one. AskVault does all of this automatically; if you use AskVault you don't implement any of it.

The failure signals to check for

Detection logic is what separates a fast scraper from a slow one. A well-built scraper escalates only when the cheap fetch shows a known-failure signal:

  • HTTP status. Anti-bot status codes such as 403 or 503 with a challenge body. A 429 means rate-limited (back off, don't hammer).
  • Body size. A suspiciously small body on a page that should be a real article usually means the content is rendered client-side.
  • Empty SPA shell. An app container (<div id="root">, __next, app, etc.) with little surrounding text. React, Next.js, Vue.
  • Challenge interstitials. Bot-protection challenge pages ("Just a moment", "Checking your browser") or a meta-refresh redirect to one.
  • Cookie wall. A consent page that blocks content until cookies are accepted.

When any of these match, escalate. Otherwise, accept the response and move on. AskVault runs this detection automatically.

DIY: a minimal Python implementation

If you're rolling your own scraper instead of using AskVault, here's the smallest version that handles both failure modes. It uses httpx for Tier 0 and a headless browser for Tier 1:

import re
import httpx
from playwright.sync_api import sync_playwright


CHALLENGE_PHRASES = (
    "just a moment", "checking your browser",
    "attention required", "enable javascript",
)
SPA_SIGNATURES = (
    '<div id="root">', '<div id="__next">', '<div id="app">',
)


def fetch_page(url: str) -> str | None:
    # Tier 0: plain HTTP
    try:
        r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0 ..."}, timeout=10, follow_redirects=True)
        if r.status_code == 200 and len(r.text) > 3000 and not any(p in r.text.lower() for p in CHALLENGE_PHRASES):
            if not any(s in r.text for s in SPA_SIGNATURES):
                return r.text
    except httpx.RequestError:
        pass
    # Tier 1: headless browser
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="networkidle", timeout=30000)
        html = page.content()
        browser.close()
        return html

A few things this minimal version doesn't do (which a production scraper should):

  • Per-host escalation cache. Once a host fails Tier 0, future URLs from that host skip Tier 0.
  • Rate limiting per host. Some sites will ban you if you fetch 50 pages per second.
  • Concurrent fetches with backoff. A real crawler runs 20+ pages in parallel with adaptive throttling.
  • Content extraction. Once you have the rendered HTML, you still need to strip nav, footer, scripts, and ads before chunking.

If you want to ship something quickly, you can take the snippet above, add a queue, and you'll have a working RAG ingestion scraper for under 100 lines of code. Just don't expect it to handle Cloudflare or hostile bot protection out of the box.

Or use AskVault

The whole point of AskVault is that we did this work. When you crawl a website through AskVault:

  • A real production scraper runs the tiered fallback automatically. No per-host configuration.
  • The per-host escalation cache learns which sites need a browser, so the second URL from a hostile host is fast.
  • Anti-bot challenges are handled at the right tier without you knowing or caring.
  • Failed crawls retry with exponential backoff so transient errors don't tank your knowledge base.
  • Extracted content goes straight into a workspace-isolated vector index ready for retrieval.

Setup: paste your URL into the onboarding wizard. Indexing 50 pages takes about 90 seconds; 500 pages takes about 10 minutes.

Detect which case applies to your site

Run a quick check before you start. It tells you whether you need a headless browser or not.

Use this one-liner: it fetches the raw HTML your server sends, before any JavaScript runs.

Terminal window
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com
curl -s https://yoursite.com | head -c 2000

Read the output. If the status is 200 and you see real content (your hero copy, navigation links, product names), you're fine. A plain HTTP scraper will work.

If you see a <div id="root"> with nothing inside, your site is JS-rendered. You need a headless browser.

If you see "Just a moment" or "Checking your browser", you're behind Cloudflare or similar. You need an anti-bot bypass service or a real browser fingerprint.

Common follow-up questions

Why not use a headless browser for everything?

Speed and resource use. A headless browser fetch takes 3 to 5 seconds, runs Chromium, and uses about 200 MB of memory per concurrent page. A plain HTTP fetch takes 200 to 800 ms and uses about 5 MB of memory.

A 1,000-page docs site crawled headless takes around 50 minutes. Crawled with HTTP-first tiering, where only the pages that actually need a browser pay the slower-fetch tax, it finishes in about 5 minutes. Roughly 10x the throughput at the same coverage.

How do I handle infinite-scroll content?

Headless browsers can scroll the page programmatically: page.evaluate("window.scrollTo(0, document.body.scrollHeight)") then wait for new content to load, repeat. AskVault's scraper handles this automatically if it detects a IntersectionObserver-style infinite-scroll pattern.

What about content behind login walls?

If the content is gated, your scraper needs the cookies of an authenticated session. AskVault supports a Cookies field in Crawl config where you paste a session cookie. The scraper sends it on every request and gets the authenticated version of the page. Be careful: log out of the source account after indexing so the cookie expires.

Will my scraping get detected and blocked?

Probably yes if you scrape aggressively without rate limits and identifying user-agent. Be polite: 1 request per second per host, identify yourself in the User-Agent, respect robots.txt. AskVault's scraper does all of this by default.

Does this work with Single-Page Apps that use client-side routing?

Yes. You need to either (a) crawl by enumerating all URLs from the sitemap, or (b) crawl the home page and discover internal links from the rendered DOM (not the raw HTML, since SPA routing renders client-side). AskVault's scraper does both. It prefers the sitemap when available.

These pages cover setup, restriction rules, and the retrieval concepts behind indexing.

Was this page helpful?