Skip to content
AskVault Docs
Try Free →

Advanced scraping settings

Written by , Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 4 min read

When to use

For sites that crawl poorly with defaults:

  • Anti-bot protection (Cloudflare, hCaptcha challenges).
  • JS-heavy rendering (React/Vue/Angular SPAs with no SSR).
  • Login-walled content.
  • Rate-limited sites that drop our requests.

Settings available

Six knobs:

  1. User-agent string. Default is AskVault-Bot/1.0. Override for sites that allowlist specific UAs.
  2. JS rendering. Toggle headless rendering on/off. Slower but handles SPAs.
  3. Retry behavior. Number of retries on 5xx, timeout window.
  4. BYOK scraper. Bring your own Firecrawl, ZenRows, or ScrapFly API key for harder sites.
  5. Per-host throttle. Concurrent fetches per origin.
  6. Custom headers. Add cookies, auth tokens (see cookies for login crawls).

JS rendering

By default, AskVault tries lightweight HTTP fetch first; falls back to headless rendering on detection of JS-required content.

Force JS rendering if your site is purely SPA-based:

  1. Settings > Advanced Scraping > Force JS Rendering.
  2. Set per source or globally.

Trade-off: 2 to 5x slower than HTTP fetch.

BYOK scraper

For aggressively-protected sites (Cloudflare Pro, custom anti-bot):

  1. Settings > BYOK Scraper > Provider.
  2. Pick provider (Firecrawl, ZenRows, or ScrapFly).
  3. Paste your API key.
  4. AskVault routes through your account for protected URLs.

You pay the upstream provider. AskVault doesn't surcharge.

Per-host throttle

For respecting your own server's capacity:

  • Default 8 in flight per host.
  • Lower to 3 if your server struggles.
  • Higher to 20 if your server has capacity.

Configure per source.

Custom headers

For sites with custom auth or rate-limit:

{
  "Authorization": "Bearer xxx",
  "X-API-Key": "yyy",
  "Cookie": "session=zzz"
}

Sent on every crawl request. See cookies for login crawls for cookies specifically.

Retry behavior

For transient errors:

  • 5xx. Retry 3 times with exponential backoff.
  • 429 rate limit. Wait per Retry-After header.
  • Timeout. 20 seconds default; configurable.

About 95% of crawls complete successfully with defaults.

Limits

  • JS rendering capacity. Headless-browser renders are memory-bounded per background worker (about 6 concurrent renders per worker, shared across whichever workspaces are crawling at that moment) rather than a fixed per-workspace allowance.
  • BYOK provider plan limits apply.
  • Custom headers count. Up to 10.
  • Retry attempts. 3 over 30 seconds.
  • Setup time. About 15 minutes.

Common pitfalls

JS render too slow. Most pages don't need it. Only force for SPAs.

BYOK provider quota. Monitor with provider; AskVault doesn't track your quota.

Throttle too tight. Crawl takes hours. Raise.

Headers leak secrets in audit log. Sensitive headers redacted.

FAQ

Will I get banned from my own site?

Configure user-agent and per-host throttle to identify and respect.

Can I use my own scraping provider's free tier?

Yes, as long as it's one of the three supported providers — Firecrawl, ZenRows, or ScrapFly. AskVault doesn't support arbitrary BYOK providers outside that list.

Does JS rendering work for React, Vue, Angular?

Yes. Headless Chromium handles all standard frameworks.

Was this page helpful?